Information processing system, server system, terminal system, information processing program, and information presentation method

ABSTRACT

An information processing system comprises a terminal system which includes a terminal device and also comprises a server system. In the terminal system, a microphone provided in the terminal device senses a sound in the vicinity thereof. Sound information representing the sound sensed by the microphone or a feature of the sound is transmitted to the server system. In the server system, the sound in the vicinity of the terminal device is specified based on the sound information. Also, presentation information to be presented by the terminal device is specified based on the specified sound, and the presentation information is transmitted to the information processing device. The terminal system receives displays the presentation information transmitted from the server, and presentation information is displayed on a display unit of the terminal device.

CROSS REFERENCE TO RELATED APPLICATION

The disclosures of Japanese Patent Application No. 2011-236765 filed onOct. 28, 2011 and Japanese Patent Application No. 2012-216015 filed onSep. 28, 2012 are incorporated herein by reference.

FIELD

The technology herein discloses an information processing system and thelike (an information processing system; and a server system, a terminalsystem, an information processing program, and an informationpresentation method which are usable in the information processingsystem) for transmitting information from a server or the like via anetwork and presenting the information on the side of a terminal.

BACKGROUND AND SUMMARY

Conventionally, there are systems for acquiring a video on the side of aterminal from a server via the Internet and displaying the video on theside of the terminal. In such a system, it is conceivable that, forexample, a game device at home acquires a video content from the serverby the Internet and causes a TV at home to display the video content.

It is not possible to present other information (e.g., advertisementinformation or additional information) to a user together with the videocontent.

Therefore, this application discloses an information processing systemand the like which can present other information to be presented to theuser together with the video content effectively. This application alsodiscloses an information processing system and the like which canpresent information useful for the user.

(1)

A non-limiting example of information processing system as describedherein comprises a terminal system including a terminal device and alsocomprises a server system.

The terminal system includes a microphone, a transmission unit, and adisplay unit. The microphone is provided in the terminal device andsenses a sound in the vicinity thereof. The transmission unit transmitssound information representing the sound sensed by the microphone or afeature of the sound to the server system. The display unit is providedin the terminal device.

The server system includes a content specification unit and apresentation information specification unit. The content specificationunit specifies a video content which is being reproduced in the vicinityof the terminal device, based on the sound information. The presentationinformation specification unit specifies presentation information to bepresented together with the specified video content, in accordance withthe video content, and transmits the presentation information to theterminal system.

The terminal system displays the presentation information transmittedfrom the server system on the display unit.

According to the configuration of (1), in the case where the videocontent is reproduced on the terminal side, the reproduction sound ofthe video content is sensed by the microphone, and the sound informationon the reproduction sound is transmitted to the server system. Theserver system specifies the video content based on the reproductionsound, and the presentation information corresponding to the specifiedvideo content is transmitted to the terminal side. Owing to this,presentation information suitable to the video content reproduced on theterminal side can be presented to the user. Namely, according to theconfiguration of (1), information useful for the user can be presented,and also other information can be presented to the user effectivelytogether with the video content.

(2)

Another example of information processing system as described hereincomprises a terminal system including a terminal device and aninformation processing device, and also comprises a server system.

The terminal device includes a microphone for sensing a sound in thevicinity thereof and a display unit. The information processing deviceincludes a transmission unit for transmitting sound informationrepresenting the sound sensed by the microphone or a feature of thesound to the server system.

The server system includes a sound specification unit and a presentationinformation specification unit. The sound specification unit specifiesthe sound in the vicinity of the terminal device based on the soundinformation. The presentation information specification unit specifiespresentation information to be presented by the terminal device inaccordance with the specified sound, and transmits the presentationinformation to the information processing device.

The information processing device receives the presentation informationtransmitted from the server system and causes the display unit of theterminal device to display the presentation information.

According to the configuration of (2), the sound on the terminal side issensed by the microphone, and sound information on the sensed sound istransmitted to the server system. The server system specifies thepresentation information corresponding to the sensed sound based on thesound information and transmits the presentation information to theterminal side. Owing to this, presentation information corresponding tothe sound on the terminal side (reproduction sound of the video content,sound of conversation of the user, etc.) can be presented to the user.Therefore, according to the configuration of (2), information useful forthe user can be presented.

(3)

The sound specification unit may specify a video content which is beingreproduced in the vicinity of the terminal device.

According to the configuration of (3), like according to theconfiguration of (1) described above, other information can be presentedto the user effectively together with the video content.

(4)

The presentation information specification unit may specify, as thepresentation information, advertisement information on a product or aservice which is regarding the video content and/or which appears in thevideo content.

According to the configuration of (4), a video content and/oradvertisement information corresponding to the specific content thevideo content can be presented to the user. Therefore, advertisementinformation can be presented to the user effectively, and thus theadvertizing effect can be improved. Since the advertisement informationis not displayed on the display device which displays the video content,the advertisement information does not block the video content displayedon the display device, and thus the video content can be presented tothe user in an easy-to-view manner.

(5)

The server system may further include a timing determination unit fordetermining a timing to present the presentation information based onthe specified video content. In this case, the terminal system displaysthe presentation information on the display unit at the determinedtiming.

According to the configuration of (5), the server system can control thetiming to present the presentation information. Owing to this, thepresentation information can be presented at an appropriate timing, andtherefore can be presented to the user effectively.

(6)

The timing determination unit may specify, based on the soundinformation, a time point of the sound sensed by the microphone on atime axis of the specified video content, and determine the timing topresent the presentation information based on the time point and a timetable of the specified video content.

According to the configuration of (6), the timing to present thepresentation information can be set easily to an appropriate timingsuitable to the specific content of the video content.

(7)

The terminal system may further include a device control unit forcontrolling a predetermined display device capable of reproducing thevideo content, regarding reproduction and/or display of the videocontent.

According to the configuration of (7), the terminal system can changethe operation of a predetermined display device. Owing to this, a videocontent and presentation information can be presented to the usereffectively.

(8)

The device control unit may be provided in the terminal device.

According to the configuration of (8), the user can operate apredetermined display device by use of the terminal device. Therefore,the ease of operation on the video content can be improved.

(9)

The server system may further include an instruction transmission unitfor specifying, based on the specified video content, a control commandregarding the reproduction and/or the display of the video content, andtransmitting the control command. In this case, the device control unitcontrols the predetermined display device based on the control commandreceived from the server system.

According to the configuration of (9), the operation of thepredetermined display device can be controlled on the server side. Forexample, the server system can control the operation of thepredetermined display device such that the video content and thepresentation information are presented to the user effectively.

(10)

The server system may further include a statistics generation unit forgenerating statistics information on the specified video contentregarding the terminal system, and transmitting the statisticsinformation to the terminal system.

According to the configuration of (10), statistics information such as,for example, a past viewing history or the like of the video content canbe presented to the user. Therefore, information useful for the user canbe presented.

(11)

The server system may further include a content transmission unit forspecifying, based on the specified video content, a sequel video contentto the video content, and transmitting the sequel video content to theterminal system. The terminal system causes a display device of theterminal device and/or a predetermined display device separate from theterminal device to display the sequel video content.

According to the configuration of (11), when the video content isterminated, the information processing system can present the sequelvideo content to the video content to the user. Owing to this,information useful for the user can be presented.

(12)

Still another example of information processing system as describedherein comprises a terminal system including a terminal device and alsocomprises a server system.

The terminal system includes a microphone, a transmission unit, and adisplay unit. The microphone is provided in the terminal device andsenses a sound in the vicinity thereof. The transmission unit transmitssound information representing the sound sensed by the microphone or afeature of the sound to the server system. The display unit is providedin the terminal device.

The server system includes a sound specification unit and a presentationinformation specification unit. The sound specification unit specifiesthe sound in the vicinity of the terminal device based on the soundinformation. The presentation information specification unit specifiespresentation information to be presented by the terminal device based onthe specified sound, and transmits the presentation information to theinformation processing device.

The terminal system receives the presentation information transmittedfrom the server system and causes the display unit of the terminaldevice to display the presentation information.

According to the configuration of (12), like according to theconfiguration of (2), information useful for the user can be presented.

This specification discloses a server system (server device) having thesame functions as those of the server system included in the informationprocessing system according to any one of (1) through (12). Thisspecification also discloses a terminal system (terminal device) havingthe same functions as those of the terminal system included in theinformation processing system. This specification further discloses aninformation processing program causing a computer to act as unitsequivalent to the units of the server system or the terminal system.This specification further discloses an information presentation methodwhich is carried out by the information processing system.

According to the information processing system and the like describedabove, information useful for the user can be presented by the terminaldevice of the terminal system.

These and other objects, features, aspects and advantages will becomemore apparent from the following detailed description when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example non-limiting video displaysystem in a first embodiment;

FIG. 2 is a block diagram showing a configuration of an examplenon-limiting transmission server in the first embodiment;

FIG. 3 is an external view of an example non-limiting game system in thefirst embodiment;

FIG. 4 is a block diagram showing an internal configuration of anexample non-limiting game device;

FIG. 5 is a perspective view showing an external configuration of anexample non-limiting controller;

FIG. 6 is a perspective view showing an external configuration of anexample non-limiting controller;

FIG. 7 is a block diagram showing an configuration of an examplenon-limiting controller;

FIG. 8 provides diagrams showing an external configuration of an examplenon-limiting terminal device;

FIG. 9 is a diagram showing an external configuration of an examplenon-limiting terminal device;

FIG. 10 is a block diagram showing an internal configuration of anexample non-limiting terminal device;

FIG. 11 shows an example of data stored on a storage unit of the gamedevice;

FIG. 12 is a main flowchart showing an example of process flowexecutable by the game device in the first embodiment;

FIG. 13 is a main flowchart showing an example of process flowexecutable by a transmission server in the first embodiment;

FIG. 14 is a flowchart showing an example of reproduction controlprocess (step 5) shown in FIG. 12 in detail;

FIG. 15 shows an example of specific content reproduced by a TV and aterminal device;

FIG. 16 shows an example of reproduction operation image;

FIG. 17 shows an example of TV operation image;

FIG. 18 is a block diagram showing an example non-limiting informationprocessing system in a second embodiment;

FIG. 19 is a block diagram showing a configuration of an examplenon-limiting server;

FIG. 20 shows an example of operation flow in the information processingsystem;

FIG. 21 shows an example of presentation timing of presentationinformation;

FIG. 22 shows an example of operation flow of the information processingsystem in the case where a control command is transmitted from theserver side;

FIG. 23 shows an example of data stored on a storage unit of the gamedevice;

FIG. 24 is a flowchart showing an example of information presentationprocess flow executed by the game device in the second embodiment;

FIG. 25 is a flowchart showing an example of presentation informationtransmission process flow executed by the server in the secondembodiment;

FIG. 26 is a flowchart showing an example of presentation informationtransmission process flow executed by the server in a modification ofthe second embodiment;

FIG. 27 is a flowchart showing an example of information presentationprocess flow executed by the game device in a modification of the secondembodiment; and

FIG. 28 is a flowchart showing an example of presentation informationtransmission process flow executed by the server in a modification ofthe second embodiment.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS FirstEmbodiment 1-1. Overall Configuration of Video Display System

Hereinafter, an example of a video display system according to a firstembodiment will be described with reference to the drawings. FIG. 1 is ablock diagram showing an example of video display system according tothe first embodiment. As shown in FIG. 1, a video display system 90includes a server-side transmission server 91 and a terminal-side gamesystem 1. The transmission server 91 and the game system 1 are mutuallycommunicable via a network 92 such as, for example, the Internet or thelike. In the first embodiment, the video display system 90 displays avideo content and advertisement information transmitted from thetransmission server 91 separately on two display devices in the gamesystem 1. Owing to this, the video content and the advertisementinformation can be presented to a user (viewer) effectively.

The transmission server 91 transmits video information to a game device3 via the network 92. In the first embodiment, the video information isinformation including at least a combination of the video content andthe advertisement information. The video content is a moving image to bepresented to the user and is displayed on a predetermined display device(in the first embodiment, TV 2) in the game system 1. The video contentmay be any content, and may be, for example, a TV program, a film or thelike. The TV program transmitted as the video content may be a programplanned to be broadcast, a program which is being broadcast, or aprogram which has been broadcast. In the first embodiment, the videocontent which has been selected by a selection instruction from the gamesystem 1 (selected by the user) is transmitted from the transmissionserver 91 to the game system 1. FIG. 1 shows one transmission server 1,but the game system 1 may be capable of accessing a plurality oftransmission servers. The video display system 90 may include aplurality of game systems 1.

The advertisement information is information including an image to bepresented to the user together with the video content. The advertisementinformation is displayed on a portable display device (in the firstembodiment, terminal device 7) in the game system 1. The advertisementinformation may be specifically any information, and typically, is animage which advertizes a product or a service. The advertisementinformation may be, for example, a moving image such as a commercialincluded in a TV program or a still image. The advertisement informationis not limited to only to a moving image or a still image, and may be animage, the displayed content of which is operated (changed) by the usersuch as, for example, an image on a web page, a game image or the like.The advertisement information may be on a product or a service which isirrelevant to the video content, or may be on the video content itself.For example, the advertisement information may be a version for sampleviewing of the video content (typically, in the case where the videocontent itself is charged).

The game system 1 receives the video information via the network 92. Thegame system 1 outputs the video content included in the received videoinformation to one of the display devices (TV 2) to be displayed, andalso outputs the advertisement information included in the videoinformation to the other portable display device (terminal device 7) tobe displayed.

The game system 1 may include any devices. In the first embodiment, asshown in FIG. 1, the game system 1 includes the game device 3 as anexample of information processing device, the TV 2 as an example ofdisplay device, and the terminal device 7 as an example of portabledisplay device. In other embodiments, the game device 3 and the TV 2 maybe integral with each other. Namely, the game system 1 may include aninformation processing device having a function of receiving the videoinformation and a function of displaying the video content. In stillother embodiments, the game device 3 and the terminal device 7 may beintegral with each other. Namely, the game system 1 may include aninformation processing device (may or may not be portable) having afunction of receiving the video information and a function of displayingthe advertisement information.

In the first embodiment, the game device 3 receives the videoinformation from the transmission server 91. The video content includedin the video information is output to the TV 2. The TV 2 displays thevideo content input from the game device 3. The advertisementinformation included in the video information is output to the terminaldevice 7. The terminal device 7 displays the advertisement informationreceived from the game device 3. In the first embodiment, thecommunication between the game device 3 and the TV 2 is wiredcommunication. In other embodiments, the communication between the gamedevice 3 and the TV 2 may be wireless communication. In the firstembodiment, the communication between the game device 3 and the terminaldevice 7 is wireless communication. In other embodiments, thecommunication between the game device 3 and the terminal device 7 may bewired communication.

As described above, in the video display system 90 according to thefirst embodiment, the video content and the advertisement informationare transmitted from the transmission server 91, and the game system 1uses two display devices to display the video content on one of thedisplay devices (TV 2) and display the advertisement information on theother display device (terminal device 7). Therefore, according to thefirst embodiment, while the video content is being reproduced by the TV2, the advertisement information can be reproduced by the terminaldevice 7. The user having the terminal device 7 can view theadvertisement information on the terminal device 7 at hand while viewingthe video content on the TV 2. According to the first embodiment, theadvertisement information can be displayed for the same time duration asthe reproduction time duration of the video content. Therefore, theadvertisement can be presented to the user effectively and thus theadvertizing effect can be improved. Also, according to the firstembodiment, the advertisement information displayed on the terminaldevice 7 is not displayed on the TV 2. Therefore, the advertisementinformation does not block the video content displayed on the TV 2, andthus the video content can be presented to the user in an easy-to-viewmanner.

1-2. Configuration of Transmission Server

Hereinafter, a configuration of the transmission server 91 will bedescribed. FIG. 2 is a block diagram showing an example of structure ofthe transmission server 91. As shown in FIG. 2, the transmission server91 includes a communication unit 101, a processing unit 102, a videostorage unit 103, and an advertisement storage unit 104. Thetransmission server 91 may have any structure which is communicable withan external device (game system 1, etc.) via the network 92 and has afunction of transmitting video information in response to an acquisitionrequest from the external device. For example, the transmission server91 includes one or more information processing devices. For example, thevideo storage unit 103 and the advertisement storage unit 104 may beincluded in different servers. In such a case, the entirety of theservers will be referred to as a “transmission server”.

The communication unit 101 is communicable with the network 92, andperforms data transmission and reception via the network 92 to and fromthe external device. Data received from the external device is output tothe processing unit 102 by the communication unit 101. Data input to thecommunication unit 101 from the processing unit 102 is transmitted tothe external device by the communication unit 101 via the network 92.

The processing unit 102 is an information processing unit for executinginformation processing in the transmission server 91. The processingunit 102 includes a CPU, a memory and the like for executing theinformation processing. In the first embodiment, the processing unit 102generates the video information in response to an acquisition requestfrom the external device (game system 1) and transmits the videoinformation to the external device via the communication unit 101.Processes executed by the processing unit 102 will be described later inmore detail.

The video storage unit 103 is a storage unit for storing (saving) dataof the video content. The video content stored on the video storage unit103 is read by the processing unit 102 when appropriate and istransmitted to the external device as the video information. Theadvertisement storage unit 104 is a storage unit for storing (saving)the advertisement information. The advertisement information stored onthe advertisement storage unit 104 is read by the processing unit 102when appropriate and is transmitted to the external device as the videoinformation.

1-3. Overall Configuration of Game System

Hereinafter, a configuration of the game system will be described. FIG.3 is an external view of an example game system. As shown in FIG. 3, agame system 1 includes a stationary display device 2 such as a TVreceiver or the like (hereinafter, referred to as the “TV 2”), astationary game device 3, an optical disc 4, a controller 5, a markerdevice 6, and a terminal device 7. In the game system 1, the game device3 executes a game process based on a game operation performed using thecontroller 5 and/or the terminal device 7, and a game image obtainedthrough the game process is displayed on the TV 2 and/or the terminaldevice 7. In other embodiments, the game system 1 may include aninformation processing device which does not have a function ofexecuting game processes, instead of the game device 3. Alternatively,the game system 1 may not include the optical disc 4, the controller 5,or the marker device 6.

In the game device 3, the optical disc 4, which is an example ofinformation storage medium used for the game device 3 in a replaceablemanner, is removably inserted. An information processing program(typically, game program) to be executed by the game device 3 is storedon the optical disc 4. The game device 3 has, on a front surfacethereof, an insertion opening for the optical disc 4. The game device 3reads and executes the information processing program stored on theoptical disc 4 which is inserted into the insertion opening, to executethe game process.

The TV 2, which is an example of predetermined display device, isconnected to the game device 3 by a connecting cord. The TV 2 is astationary display device having a larger screen than a display unit(LCD 51) of the terminal device 7. The TV 2 displays an image outputfrom the game device 3. The TV 2 includes a speaker 2 a (FIG. 4) foroutputting a sound which is output from the game device 3.

The marker device 6 is provided along a periphery of the screen (on theupper side with respect to the screen in FIG. 3) of the TV 2. A user(player) can perform a game operation by moving the controller 5, thedetails of which will be described later, and the marker device 6 isused by the game device 3 for calculating the movement, position,attitude, etc. of the controller 5. The marker device 6 includes twomarkers 6R and 6L on opposite ends thereof. Specifically, the marker 6R(as well as the marker 6L) includes one or more infrared LEDs (LightEmitting Diodes), and emits infrared light in a forward direction fromthe TV 2. The marker device 6 is connected to the game device 3 in awired connection (or a wireless connection), and the game device 3 cancontrol lighting of each infrared LED of the marker device 6. The markerdevice 6 is portable so that the user can install the marker device 6 inany desired position. While FIG. 3 shows an embodiment in which themarker device 6 is arranged on top of the TV 2, the position and thedirection of arranging the marker device 6 are not limited to thisparticular arrangement.

The controller 5 provides the game device 3 with operation datarepresenting a content of an operation performed on the controller 5itself. The controller 5 and the game device 3 can wirelesslycommunicate with each other. In other embodiments, the controller 5 andthe game device 3 may be connected in a wired manner. While FIG. 3 showsonly one controller 5 included in the game system 1, the game system 1may include a plurality of controllers 5.

The terminal device 7 is portable and is of a size that can be held bythe user. In use, the user can hold and move the terminal device 7 orcan place the terminal device 7 in any desired position. As will bedescribed in detail later, the terminal device 7 includes an LCD (LiquidCrystal Display) 51, and an input unit (touch panel 52, gyrosensor 74,etc. to be described later). The terminal device 7 can communicate withthe game device 3 wirelessly (or in a wired manner). The terminal device7 receives data on an image (e.g., advertisement image) transmitted fromthe game device 3, and displays the image on the LCD 51. In the firstembodiment, the LCD is used as the display device, but the terminaldevice 7 may include any other display device such as a display deviceutilizing EL (Electro Luminescence) or the like. The terminal device 7also transmits operation data representing a content of an operationperformed thereon to the game device 3.

1-4. Internal Configuration of Game Device 3

Now, an internal configuration of the game device 3 will be describedwith reference to FIG. 4. FIG. 4 is a block diagram illustrating aninternal configuration of the example game device 3. The game device 3includes a CPU (Central Processing Unit) 10, a system LSI 11, anexternal main memory 12, a ROM/RTC 13, a disc drive 14, an AV-IC 15, andthe like.

The CPU 10 executes a game process by executing a game program stored onthe optical disc 4, and functions as a game processor. The CPU 10 isconnected to the system LSI 11. The external main memory 12, the ROM/RTC13, the disc drive 14, and the AV-IC 15, as well as the CPU 10, areconnected to the system LSI 11. The system LSI 11 performs a process ofcontrolling data transmission between the respective componentsconnected thereto, generating an image to be displayed, obtaining datafrom an external device(s), or the like. The internal configuration ofthe system LSI 11 will be described below. The external main memory 12is of a volatile type and stores a program such as a game program readfrom the optical disc 4, a game program read from a flash memory 17 orthe like, and various data. The external main memory 12 is used as awork area or a buffer area for the CPU 10. The ROM/RTC 13 includes a ROM(so-called boot ROM) incorporating a boot program for the game device 3,and a clock circuit (RTC: Real Time Clock) for counting time. The discdrive 14 reads program data, texture data, or the like from the opticaldisc 4, and writes the read data into an internal main memory 11 e (tobe described below) or the external main memory 12.

The system LSI 11 includes an input/output processor (I/O processor) 11a, a GPU (Graphics Processor Unit) 11 b, a DSP (Digital SignalProcessor) 11 c, VRAM (Video RAM) 11 d, and the internal main memory 11e. Although not shown in the figures, these components 11 a through 11 eare connected with each other through an internal bus.

The GPU 11 b, acting as a part of a rendering mechanism, generates animage in accordance with a graphics command (rendering instruction) fromthe CPU 10. The VRAM 11 d stores data (data such as polygon data,texture data or the like) used by the GPU 11 b to execute a graphicscommand. For generating an image, the GPU 11 b generates image datausing data stored on the VRAM 11 d. In the first embodiment, the gamedevice 3 generates both of an image to be displayed on the TV 2 and animage to be displayed on the terminal device 7. Hereinafter, the imageto be displayed on the TV 2 may be referred to as a “TV image” and theimage to be displayed on the terminal device 7 may be referred to as a“terminal image”.

The DSP 11 c, functioning as an audio processor, generates sound datausing sound data or sound waveform (e.g., tone quality) data stored onthe internal main memory 11 e or the external main memory 12. In thefirst embodiment, sounds (game sounds) to be generated are classifiedinto two types as in the case of the game images, one being output bythe speaker of the TV 2 and the other being output by speakers of theterminal device 7. Hereinafter, the sound to be output by the TV 2 maybe referred to as a “TV sound”, and the sound to be output by theterminal device 7 may be referred to as a “terminal sound”.

Among the images and sounds generated by the game device 3 as describedabove, data on the image and data on the sound to be output by the TV 2are read out by the AV-IC 15. The AV-IC 15 outputs the read-out imagedata to the TV 2 via an AV connector 16, and outputs the read-out sounddata to the speaker 2 a built in the TV 2. Thus, the image is displayedon the TV 2, and the sound is output by the speaker 2 a. While theconnection scheme between the game device 3 and the TV 2 may be anyscheme, the game device 3 may transmit a control command for controllingthe TV 2 to the TV 2 via a wired connection or a wireless connection.For example, an HDMI (High-Definition Multimedia Interface) cable inconformity with the HDMI standard may be used. According to the HDMIstandard, it is possible to control the connected device by a functioncalled CEC (Consumer Electronics Control). Thus, in the case where thegame device 3 can control the TV 2, as when an HDMI cable is used, thegame device 3 can turn on the power of the TV 2 or switch the input ofthe TV 2 from one to another at an appropriate timing.

Among the images and sounds generated by the game device 3, data on theimage and data on the sound to be output by the terminal device 7 aretransmitted to the terminal device 7 by the input/output processor 11 a,etc. The data transmission to the terminal device 7 by the input/outputprocessor 11 a, etc., will be described later.

The input/output processor 11 a exchanges data with components connectedthereto, and downloads data from an external device(s). The input/outputprocessor 11 a is connected to the flash memory 17, a networkcommunication module 18, a controller communication module 19, anextension connector 20, a memory card connector 21, and a codec LSI 27.An antenna 22 is connected to the network communication module 18. Anantenna 23 is connected to the controller communication module 19. Thecodec LSI 27 is connected to a terminal communication module 28, and anantenna 29 is connected to the terminal communication module 28.

The game device 3 can be connected to the network 92 such as theInternet or the like to communicate with an external informationprocessing device (e.g., any of various types of servers such as thetransmission server 91 or the like). Specifically, the input/outputprocessor 11 a can be connected to a network such as the Internet or thelike via the network communication module 18 and the antenna 22, and cancommunicate with other information processing devices connected to thenetwork. The input/output processor 11 a regularly accesses the flashmemory 17, and detects the presence or absence of any data to betransmitted to the network. When such data is detected, the input/outputprocessor 11 a transmits the data to the network 92 via the networkcommunication module 18 and the antenna 22. Further, the input/outputprocessor 11 a receives data transmitted from an external informationprocessing device and data downloaded from a download server, via thenetwork, via the antenna 22 and via the network communication module 18,and stores the received data on the flash memory 17. The CPU 10 executesa game program to read data stored on the flash memory 17 and to use thedata in the game program. The flash memory 17 may store saved game data(e.g., game result data or unfinished game data) of a game played usingthe game device 3 in addition to data exchanged between the game device3 and an external information processing device. The flash memory 17 mayalso have a game program stored thereon.

The game device 3 can receive operation data from the controller 5.Specifically, the input/output processor 11 a receives operation datatransmitted from the controller 5 via the antenna 23 and the controllercommunication module 19, and stores (temporarily stores) the operationdata in a buffer area of the internal main memory 11 e or the externalmain memory 12.

The game device 3 can exchange image data, sound data or any other datawith the terminal device 7. For transmitting an image (terminal image)to the terminal device 7, the input/output processor 11 a outputs dataon an image generated by the GPU 11 b to the codec LSI 27. The codec LSI27 performs a predetermined compression process on the image data fromthe input/output processor 11 a. The terminal communication module 28wirelessly communicates with the terminal device 7. Accordingly, theimage data compressed by the codec LSI 27 is transmitted by the terminalcommunication module 28 to the terminal device 7 via the antenna 29. Thetransmission of the image data from the game device 3 to the terminaldevice 7 may be performed such that delay is avoided as much aspossible. Therefore, in the first embodiment, the codec LSI 27compresses the image data using a compression technology with highefficiency such as, for example, the H.264 standard. Other compressiontechnologies may be used, or the image data may be transmitteduncompressed if the communication speed is sufficiently high. Theterminal communication module 28 is, for example, a Wi-Fi certifiedcommunication module, and may perform wireless communication at highspeed with the terminal device 7 using a MIMO (Multiple Input MultipleOutput) technology employed in, for example, the IEEE 802.11n standard,or may use other communication schemes.

In addition to the image data, the game device 3 also transmits sounddata to the terminal device 7. Specifically, the input/output processor11 a outputs the sound data generated by the DSP 11 c to the terminalcommunication module 28 via the codec LSI 27. The codec LSI 27 performsa compression process on the sound data as it does on the image data.The method of compressing the sound data may be any method. In otherembodiments, the sound data may be transmitted uncompressed. Theterminal communication module 28 transmits such compressed image dataand sound data to the terminal device 7 via the antenna 29.

In addition to the image data and the sound data, the game device 3transmits various types of control data (referred to as “terminalcontrol data”) to the terminal device 7 when necessary. The terminalcontrol data is data representing an instruction to control a componentincluded in the terminal device 7, for example, an instruction tocontrol lighting of a marker unit (marker unit 55 shown in FIG. 10) oran instruction to control image capturing by a camera (camera 56 shownin FIG. 10). The input/output processor 11 a transmits the terminalcontrol data to the terminal device 7 in accordance with an instructionfrom the CPU 10. In the first embodiment, the codec LSI 27 does notperform a compression process on the terminal control data, but in otherembodiments, the codec LSI 27 may perform a compression process. Thedata to be transmitted from the game device 3 to the terminal device 7may be encrypted when necessary or may not be encrypted.

The game device 3 can receive various types of data from the terminaldevice 7. As will be described in detail later, in the first embodiment,the terminal device 7 transmits operation data, image data, and sounddata. The data transmitted by the terminal device 7 is received by theterminal communication module 28 via the antenna 29. Here, the imagedata and the sound data from the terminal device 7 have been subjectedto the same compression process as performed on the image data and thesound data transmitted from the game device 3 to the terminal device 7.Accordingly, the image data and the sound data are transferred from theterminal communication module 28 to the codec LSI 27, and subjected to adecompression process by the codec LSI 27 before being output to theinput/output processor 11 a. Meanwhile, the operation data from theterminal device 7 is smaller in size than the image data or the sounddata and therefore may not be subjected to a compression process. Theoperation data may be encrypted when necessary or may not be encrypted.Accordingly, after being received by the terminal communication module28, the operation data is output to the input/output processor 11 a viathe codec LSI 27. The input/output processor 11 a stores (temporarilystores) the data received from the terminal device 7 in a buffer area ofthe internal main memory 11 e or the external main memory 12.

The game device 3 can be connected to other devices or external storagemediums. Specifically, the input/output processor 11 a is connected tothe extension connector 20 and the memory card connector 21. Theextension connector 20 is a connector for an interface, such as a USB orSCSI interface. The extension connector 20 can be connected to a mediumsuch as an external storage medium, a peripheral device such as anothercontroller, or a wired communication connector, and thus can communicatewith a network in place of the network communication module 18. Thememory card connector 21 is a connector for connecting thereto anexternal storage medium such as a memory card. For example, theinput/output processor 11 a can access an external storage medium viathe extension connector 20 or the memory card connector 21 to store dataon the external storage medium or read data from the external storagemedium.

The game device 3 includes a power button 24, a reset button 25, and aneject button 26. The power button 24 and the reset button 25 areconnected to the system LSI 11. When the power button 24 is turned on,power is supplied from an external power source to the components of thegame device 3 via an AC adaptor (not shown). When the reset button 25 ispressed, the system LSI 11 reboots a boot program of the game device 3.The eject button 26 is connected to the disc drive 14. When the ejectbutton 26 is pressed, the optical disc 4 is ejected from the disc drive14.

In other embodiments, some of the components of the game device 3 may beprovided as extension devices separate from the game device 3. In thiscase, an extension device may be connected to the game device 3 via, forexample, the extension connector 20. Specifically, an extension devicemay include, for example, the codec LSI 27, the terminal communicationmodule 28, and the antenna 29, and can be attached to or detached fromthe extension connector 20. Thus, by connecting the extension device toa game device which does not include the above components, the gamedevice can communicate with the terminal device 7.

1-5. Configuration of Controller 5

Now, with reference to FIGS. 5 through 7, the controller 5 will bedescribed. FIG. 5 is a perspective view illustrating an externalconfiguration of the example controller 5. FIG. 6 is a perspective viewillustrating an external configuration of the controller 5. Theperspective view of FIG. 5 shows the controller 5 as viewed from the toprear side thereof, and the perspective view of FIG. 6 shows thecontroller 5 as viewed from the bottom front side thereof. Thecontroller 5 may be any controller device capable of being controlled bythe player. In the first embodiment, the controller 5 includes buttonsand sensors, and the player can perform button operations and anoperation of moving the controller device itself. While the controller 5does not include a display unit in the first embodiment, the controller5 may include a display unit in other embodiments.

As shown in FIG. 5 and FIG. 6, the controller 5 includes a housing 31formed by, for example, plastic molding. The housing 31 has a generallyparallelepiped shape extending in a longitudinal direction from front torear (Z-axis direction shown in FIG. 5), and as a whole is sized to beheld by one hand of an adult or even a child. The user can perform anoperation by pressing a button provided in the controller 5, and movingthe controller 5 itself to change the position and the attitude (tilt)thereof.

The housing 31 has a plurality of operation buttons. As shown in FIG. 5,on a top surface of the housing 31, a cross button 32 a, a first button32 b, a second button 32 c, an A button 32 d, a minus button 32 e, ahome button 32 f, a plus button 32 g, and a power button 32 h areprovided. In the example embodiment, the top surface of the housing 31on which the buttons 32 a through 32 h are provided may be referred toas a “button surface”. Meanwhile, as shown in FIG. 6, a recessed portionis formed in a bottom surface of the housing 31, and a B button 32 i isprovided on a rear slope surface of the recessed portion. The operationbuttons 32 a through 32 i are appropriately assigned respectivefunctions in accordance with the information processing program to beexecuted by the game device 3. The power button 32 h is intended toremotely turn ON/OFF the game device 3.

On a rear surface of the housing 31, the connector 33 is provided. Theconnector 33 is used for connecting the controller 5 to another device(e.g., another sensor unit or controller). Both sides of the connector33 on the rear surface of the housing 31 have an engagement hole 33 afor preventing easy inadvertent disengagement of the above-describedanother device.

In a rear-side portion of the top surface of the housing 31, a pluralityof (four in FIG. 5) LEDs 34 a through 34 d are provided. The controller5 is assigned a controller type (number) so as to be distinguishablefrom another controller. The LEDs 34 a through 34 d are each used, forexample, for informing the user of the controller type which iscurrently set for the controller 5, or for informing the user ofremaining battery power of the controller 5. Specifically, when anoperation is to be performed using the controller 5, one of the LEDs 34a through 34 d corresponding to the controller type is lit up.

The controller 5 has an image-capturing/processing unit 35 (FIG. 7), anda light incident surface 35 a through which light is incident on theimage-capturing/processing unit 35 is provided on a front surface of thehousing 31 as shown in FIG. 6. The light incident surface 35 a is formedof a material transmissive of at least infrared light output by themarkers 6R and 6L.

On the top surface of the housing 31, sound holes 31 a for externallyoutputting a sound from a speaker (not shown) incorporated in thecontroller 5 is provided between the first button 32 b and the homebutton 32 f.

A substrate is fixed inside the housing 31, and the operation buttons 32a through 32 h, the LEDs 34 a through 34 d, an acceleration sensor 37,an antenna 45, a speaker, a wireless module 44, and the like areprovided on the substrate. These components are connected to amicrocomputer 42 (FIG. 7) via lines (not shown) formed on the substrateor the like.

At a front edge of the substrate, the image-capturing/processing unit 35is provided. The image-capturing/processing unit 35 includes an infraredfilter 38, a lens 39, an image-capturing element 40 and an imageprocessing circuit 41 (FIG. 7) located in this order from the front sideof the controller 5. These components 38 through 41 are attached on abottom main surface of the substrate.

Moreover on the substrate, the microcomputer 42 and a vibrator areprovided. The vibrator is, for example, a vibration motor or a solenoid,and is connected to the microcomputer 42 via lines formed on thesubstrate or the like. The controller 5 is vibrated by actuation of thevibrator based on a command from the microcomputer 42. Thus, thevibration is conveyed to the user's hand holding the controller 5, andthus a so-called vibration-feedback game is realized.

In addition to the above components, the controller 5 includes a crystaloscillator for generating a basic clock for the microcomputer 42, anamplifier for outputting a sound signal to the speaker, etc.

The shape of the controller 5, the shape of each operation button, thenumber and the positions of acceleration sensor and vibrator and so onshown in FIGS. 5 and 6 are merely examples, and other shapes, numbers,and positions may be employed. Although in the first embodiment, theimaging direction of an image-capturing mechanism is a Z-axis positivedirection, the imaging direction may be any direction. That is, theimage-capturing/processing unit 35 (light incident surface 35 a throughwhich light is incident on the image-capturing/processing unit 35) ofthe controller 5 may not necessarily be provided on the front surface ofthe housing 31, but may be provided on any other surface on which lightcan be received from the outside of the housing 31.

FIG. 7 is a block diagram illustrating a configuration of the examplecontroller 5. The controller 5 includes the operating units 32(operation buttons 32 a through 32 i), the image-capturing/processingunit 35, a communication unit 36, the acceleration sensor 37, and agyrosensor 48. The controller 5 transmits, to the game device 3, datarepresenting the content of an operation performed on the controller 5itself as operation data. Hereinafter, operation data transmitted by thecontroller 5 may be referred to as “controller operation data”, andoperation data transmitted by the terminal device 7 may be referred toas “terminal operation data”.

The operating unit 32 includes the operation buttons 32 a through 32 idescribed above, and outputs, to the microcomputer 42 of thecommunication unit 36, operation button data representing an input stateof each of the operation buttons 32 a through 32 i (that is, whether ornot each operation button 32 a through 32 i has been pressed).

The image-capturing/processing unit 35 is a system for analyzing imagedata taken by the image-capturing mechanism and calculating, forexample, the centroid, the size or the like of an area having a highbrightness in the image data. The image-capturing/processing unit 35includes the infrared filter 38, the lens 39, the image-capturingelement 40 and the image processing circuit 41. The infrared filter 38transmits therethrough only infrared light included in the lightincident on a front surface of the controller 5. The lens 39 collectsthe infrared light transmitted through the infrared filter 38 so as tobe incident on the image-capturing element 40. The image-capturingelement 40 is a solid-state imaging device such as, for example, a CMOSsensor or a CCD sensor, which receives the infrared light collected bythe lens 39 and outputs an image signal. The marker unit 55 of theterminal device 7 and the marker device 6, which are image-capturingtargets, each include a marker(s) for outputting infrared light.Therefore, the infrared filter 38 enables the image-capturing element 40to receive only the infrared light transmitted through the infraredfilter 38 and generate image data, so that an image of eachimage-capturing target (marker unit 55 and/or marker device 6) can becaptured with accuracy. Hereinafter, an image captured by theimage-capturing element 40 will be referred to as a “captured image”.The image data generated by the image-capturing element 40 is processedby the image processing circuit 41. The image processing circuit 41calculates a position, in the captured image, of an image-capturingtarget. The image processing circuit 41 outputs a coordinate point ofthe calculated position to the microcomputer 42 of the communicationunit 36. Data representing the coordinate point is transmitted asoperation data to the game device 3 by the microcomputer 42.Hereinafter, such a coordinate point will be referred to as a “markercoordinate point”. The marker coordinate point changes depending on theattitude (angle of tilt) or the position of the controller 5 itself, andtherefore the game device 3 can calculate the attitude and the positionof the controller 5 using the marker coordinate point.

In other embodiments, the controller 5 may not include the imageprocessing circuit 41, and the controller 5 may transmit the capturedimage as it is to the game device 3. In this case, the game device 3 mayinclude a circuit or a program having the same function as that of theimage processing circuit 41 and calculate the marker coordinate point.

The acceleration sensor 37 detects an acceleration (including agravitational acceleration) of the controller 5, that is, a force(including gravity) applied to the controller 5. The acceleration sensor37 detects a value of an acceleration (linear acceleration) applied to adetection unit of the acceleration sensor 37 in a straight linedirection along a sensing axis direction, among all accelerationsapplied to a detection unit of the acceleration sensor 37. While theacceleration sensor 37 may be any acceleration sensor capable ofdetecting an acceleration in one or more sensing axis directions, theacceleration sensor 37 in the first embodiment detects a linearacceleration in three axis directions with respect to the controller 5,i.e., an up-down direction (Y-axis direction shown in FIG. 5), aleft-right direction (X-axis direction shown in FIG. 5), and afront-rear direction (Z-axis direction shown in FIG. 5). Theacceleration sensor 37 is, for example, a capacitive MEMS (Micro-ElectroMechanical System) acceleration sensor. However, another type ofacceleration sensor may be used. Data representing the acceleration(acceleration data) detected by the acceleration sensor 37 is output tothe communication unit 36. The acceleration detected by the accelerationsensor 37 changes depending on the attitude (angle of tilt) or themovement of the controller 5, and therefore the game device 3 cancalculate the attitude and the movement of the controller 5 using theobtained acceleration data. In the first embodiment, the game device 3calculates the attitude, angle of tilt, etc., of the controller 5 basedon the obtained acceleration data.

The gyrosensor 48 detects angular velocities about three axes (in thefirst embodiment, the X-, Y-, and Z-axes). In this specification, thedirections of rotation about the X-axis, the Y-axis, and the Z-axisrelative to the imaging direction (Z-axis positive direction) of thecontroller 5 are referred to as a pitch direction, a yaw direction, anda roll direction, respectively. So long as the gyrosensor 48 can detectthe angular velocities about the three axes, any number of gyrosensorsor any combination of gyrosensors may be included therein. Datarepresenting the angular velocities detected by the gyrosensor 48 isoutput to the communication unit 36. Alternatively, the gyrosensor 48may detect an angular velocity about one axis or angular velocitiesabout two axes.

The communication unit 36 includes the microcomputer 42, a memory 43,the wireless module 44 and the antenna 45. By the communication unit 36(wireless module 44 and antenna 45), the controller 5 functions as awireless controller. The microcomputer 42 controls the wireless module44 for wirelessly transmitting data obtained by the microcomputer 42 tothe game device 3, while using the memory 43 as a storage area forperforming a process.

Data output by each of the operating units 32, theimage-capturing/processing unit 35, the acceleration sensor 37, and thegyrosensor 48 to the microcomputer 42 is temporarily stored on thememory 43. These types of data are transmitted to the game device 3 asoperation data (controller operation data). Specifically, at the time oftransmission to the controller communication module 19 of the gamedevice 3, the microcomputer 42 outputs the operation data stored on thememory 43 to the wireless module 44. The wireless module 44 uses, forexample, the Bluetooth (registered trademark) technology to transmit theoperation data to the game device 3. The CPU 10 of the game device 3performs the game process using the operation data obtained from thecontroller 5. The wireless transmission from the communication unit 36to the controller communication module 19 is sequentially performed at apredetermined time interval. Since the game process is generallyperformed at a cycle of 1/60 sec. (corresponding to one frame time),data may be transmitted at a cycle of a shorter time period.

As described above, the controller 5 can transmit each of markercoordinate data, acceleration data, angular velocity data, and operationbutton data as operation data representing an operation performedthereon. In addition, the game device 3 executes the game process usingthe operation data as a game input. Accordingly, by using the controller5, the user can perform the game operation of moving the controller 5itself, in addition to a conventional general game operation of pressingan operation button. For example, the user can perform operations oftilting the controller 5 to an arbitrary attitude, pointing thecontroller 5 to an arbitrary position on the screen, and moving thecontroller 5 itself.

1-6. Configuration of Terminal Device 7

Now, a configuration of the terminal device 7 will be described withreference to FIGS. 8 through 10. FIG. 8 is a plan view showing anexternal configuration of the example terminal device 7. FIG. 8( a) is afront view of the terminal device 7, FIG. 8( b) is a top view thereof,FIG. 8( c) is a right side view thereof, and FIG. 8( d) is a bottom viewthereof. FIG. 9 is a rear view of the example terminal device 7. Theterminal device 7 may be of any configuration as long as including adisplay unit.

As shown in FIG. 8, the terminal device 7 includes a housing 50generally having a horizontally-elongated rectangular plate shape. Itcan also be considered that the terminal device 7 is a tablet-typeinformation processing device. The housing 50 may have a curved surfaceor may have a protrusion, etc., as long as it is generally in a plateshape. The housing 50 is sized so as to be held by the user. Thus, theuser can hold and move the terminal device 7, or can change the positionin which the terminal device 7 is placed. Although the details will bedescribed later, the terminal device 7 is configured so as to be heldand operated easily by the user even though the terminal device 7 issuch a relatively large terminal device (controller device) as describedabove.

The terminal device 7 includes the LCD 51 on a front surface (frontside) of the housing 50. The LCD 51 is provided near the center of thefront surface of the housing 50. Therefore, the user can hold and movethe terminal device 7 while looking at a screen of the LCD 51 by holdingopposing end portions of the housing 50 with respect to the LCD 51. Theuser can hold the terminal device 7 in a landscape position (in ahorizontally-oriented direction) by holding left and right opposing endportions of the housing 50 with respect to the LCD 51, or the user canhold the terminal device 7 in a portrait position (in avertically-oriented direction).

As shown in FIG. 8( a), the terminal device 7 includes the touch panel52 on the screen of the LCD 51 as an operating unit. The touch panel 52may be of any type, for example, a resistive type, a capacitive type,etc. The touch panel 52 may be of a single-touch type or a multi-touchtype. An input on the touch panel 52 can be made with a touch pen 60, afinger of the user, etc. The housing 50 has an accommodation hole 60 afor accommodating the touch pen 60 usable for performing an operation onthe touch panel 52 (see FIG. 8( b)). The accommodation hole 60 a isprovided in a top surface of the housing 50 so that the touch pen 60does not fall here. Alternatively, the accommodation hole 60 a may beprovided on a side surface or a bottom surface of the housing 50.

As shown in FIG. 8, the terminal device 7 includes two analog sticks(slide pads) 53A and 53B and a plurality of buttons (keys) 54A through54M, as operating units. The analog sticks 53A and 53B are each adirection-specifying device. The analog sticks 53A and 53B are eachconfigured so that a movable member (stick portion) can be slid in anydirection (at any angle in the up, down, left, right and diagonaldirections) with respect to the front surface of the housing 50. Themovable member of each of the analog sticks 53A and 53B may be of such atype as to be tilted in any direction with respect to the front surfaceof the housing 50. The left analog stick 53A is provided on the leftside of the screen of the LCD 51, and the right analog stick 53B isprovided on the right side of the screen of the LCD 51.

The buttons 54A through 54L are operating units for making predeterminedinputs, and are keys that can be pressed. As will be discussed below,the buttons 54A through 54L are provided at such positions that the usercan operate them while holding left and right portions of the terminaldevice 7. Therefore, the user can operate these operating units easilyeven when holding and moving the terminal device 7.

As shown in FIG. 8( a), the cross button (direction-input button) 54Aand the buttons 54B through 54H and 54M, among the operation buttons 54Athrough 54M, are provided on a front surface of the housing 50. Thesebuttons 54A through 54H and 54M are provided at positions at which thesebuttons can be operated by the thumbs of the user. The cross button 54Ais provided on the left side of the LCD 51 and below the left analogstick 53A. The cross button 54A has a cross shape, and is a button withwhich it is possible to specify at least up, down, left and rightdirections.

The buttons 54B through 54D are provided below the LCD 51. These threebuttons 54B through 54D are provided at positions at which these buttonscan be operated with either the left or the right hand of the user. Theterminal device 7 includes the power button 54M for turning ON/OFF thepower of the terminal device 7. The power of the game device 3 can beremotely turned ON/OFF by operating the power button 54M. The powerbutton 54M is provided below the LCD 51 as are the buttons 54B through54D. The power button 54M is provided on the right side of the buttons54B through 54D. The four buttons 54E through 54H are provided on theright side of the LCD 51 and below the right analog stick 53B. The fourbuttons 54E through 54H are also provided at upper, lower, left andright positions (with respect to the center position among the fourbuttons 54E through 54H). Therefore, with the terminal device 7, thefour buttons 54E through 54H can also serve as buttons with which theuser specifies the up, down, left and right directions.

In the first embodiment, a projecting portion (eaves portion 59) isprovided on the rear side of the housing 50 (side opposite to the frontsurface where the LCD 51 is provided) (see FIGS. 8( c) and 9). As shownin FIG. 8( c), the eaves portion 59 is a mountain-shaped member which isprojecting from a rear surface of the generally plate-shaped housing 50.The projecting portion has such a height (thickness) that it can rest onfingers of the user holding the rear surface of the housing 50. The usercan hold the terminal device 7 stably without getting tired even if theterminal device 7 has a relatively large size, by holding the terminaldevice 7 while resting his/her fingers on the eaves portion 59 (placingthe eaves portion 59 on the fingers). That is, the eaves portion 59 canbe referred to as a supporting member for allowing the fingers tosupport the housing 50, or can be referred to also as a “finger-restingportion”.

The eaves portion 59 is provided above the center of the housing 50 withrespect to an up-down direction. The eaves portion 59 is provided on thereverse side so as to generally correspond to the operating units(analog sticks 53A and 53B) which are provided on the front surface ofthe housing 50. That is, the projecting portion is provided so as toextend across an area, on the reverse side, including positionscorresponding to the operating units which are provided respectively onthe left side and on the right side of the display unit. Therefore, foroperating the operating units, the user can hold the terminal device 7so as to support the eaves portion 59 with his/her middle fingers orring fingers. Thus, it is easy to hold the terminal device 7, and it iseasy to operate the operating units. In the first embodiment, since theprojecting portion has an eaves-like shape extending in a left-rightdirection, the user can hold the terminal device 7 with his/her middlefingers or ring fingers placed along a bottom surface of the projectingportion. This makes it easy to hold the terminal device 7. In otherembodiments, the eaves portion 59 may be located at any position and mayhave any size and any shape. The eaves portion 59 may not be formed.

As shown in FIGS. 8( a), 8(b) and 8(c), a first L button 54I and a firstR button 54J are provided respectively in left and right opposingportions on the top surface of the housing 50. In other embodiments, theoperating units provided respectively in the left and right portions onthe top surface of the housing 50 do not have to be provided at left andright ends, and may be provided at positions other than the ends. Theoperating units may be provided respectively on left and right sidesurfaces of the housing 50. As shown in FIGS. 8( c) and 9, a second Lbutton 54K and a second R button 54L are provided on a top surface ofthe projecting portion (eaves portion 59). The second L button 54K isprovided near a left end of the eaves portion 59. The second R button54L is provided near a right end of the eaves portion 59.

In the first embodiment, the user can hold the terminal device 7 easilyby holding the terminal device 7 with his/her fingers abutting againstthe bottom surface of the projecting portion (eaves portion 59). Sincethe first L button 54I and the first R button 54J are provided on thetop surface of the housing 50 and the second L button 54K and the secondR button 54L are provided on the top surface of the projecting portion,the user can operate these buttons easily in the state described above.The user can hold the terminal device 7 with his/her ring fingersabutting against the bottom surface of the eaves portion 59 (so as tosupport the eaves portion 59 with his/her ring fingers), or can alsohold the terminal device 7 with his/her middle fingers abutting againstthe bottom surface of the eaves portion 59.

With the terminal device 7 in the first embodiment, the projectingportion (eaves portion 59) is provided on the rear surface. Therefore,when the terminal device 7 is put down with the screen of the LCD 51(front surface of the housing 50) facing up, the screen is slightlyinclined. Therefore, the screen is seen easily with the terminal device7 put down. Input operations to the touch panel 52 are performed easilywith the terminal device 7 put down. In other embodiments, an additionalprojecting portion having generally the same height as the eaves portion59 may be formed on the rear surface of the housing 50. In this case,with the screen of the LCD 51 facing up, the terminal device 7 can beput down so that the screen is horizontal with the projecting portionsin contact with a floor surface. The additional projecting portion maybe a removable (or foldable) member. In this case, the terminal devicecan be put down with the screen either slightly inclined or with thescreen horizontal. That is, in the case where the terminal device 7 isput down and used, the eaves portion 59 can be used as a leg portion.

The buttons 54A through 54L are each assigned an appropriate function inaccordance with the game program. For example, the cross button 54A andthe buttons 54E through 54H may be used for direction-specifyingoperations, selection operations, etc., whereas the buttons 54B through54E may be used for OK button operations, cancel button operations, etc.The terminal device 7 may include a button for turning ON/OFF the powerof the LCD 51, or a button for performing a connection setting (pairing)with the game device 3.

As shown in FIG. 8( a), the terminal device 7 includes the marker unit55 including the marker 55A and the marker 55B on the front surface ofthe housing 50. The marker unit 55 is provided above the LCD 51. Themarker 55A and the marker 55B are each formed by one or more infraredLEDs, as are the markers 6R and 6L of the marker device 6. The infraredLEDs of the markers 55A and 55B are provided inside a window portionthat is transmissive of infrared light. The marker unit 55 is used forthe game device 3 to calculate the movement, etc., of the controller 5,as is the marker device 6 described above. The game device 3 can controllighting of the infrared LEDs of the marker unit 55.

The terminal device 7 includes the camera 56 as the image-capturingmechanism. The camera 56 includes an image-capturing element (e.g., CCDimage sensor, CMOS image sensor, or the like) having a predeterminedresolution, and a lens. As shown in FIG. 8, the camera 56 is provided onthe front surface of the housing 50 in the first embodiment. Therefore,the camera 56 can capture an image of the face of the user holding theterminal device 7, and can capture an image of the user playing a gamewhile looking at the LCD 51, for example. In the first embodiment, thecamera 56 is provided between the two markers 55A and 55B.

The terminal device 7 includes a microphone 79 as a sound inputmechanism. A microphone hole 50 c is provided in the front surface ofthe housing 50. The microphone 79 is provided inside the housing 50behind the microphone hole 50 c. The microphone 79 detects a soundaround the terminal device 7 such as the voice of the user or the like.

The terminal device 7 includes speakers 77 as the sound outputmechanism. As shown in FIG. 8( d), speaker holes 57 are provided in alower portion of the front surface of the housing 50. An output soundfrom the speakers 77 is output from the speaker holes 57. In the firstembodiment, the terminal device 7 includes two speakers, and the speakerholes 57 are provided at the respective positions of the left speakerand the right speaker. The terminal device 7 includes a knob 64 foradjusting the sound volume of the speakers 77. The terminal device 7includes a sound output terminal 62 connectable to a sound output unitsuch as an earphone. Although the sound output terminal 62 and the knob64 are provided on the top surface of the housing 50 considering thefact that an additional device is connected to the bottom surface of thehousing 50, the sound output terminal 62 and the knob 64 mayalternatively be provided on the left or right side surface or on thebottom surface.

The housing 50 includes a window 63 through which an infrared signalfrom an infrared communication module 82 is output to the outside of theterminal device 7. The window 63 is provided on the top surface of thehousing 50 so that the infrared signal is output in a forward directionof the user when the user holds the opposing sides of the LCD 51. Inother embodiments, the window 63 may be provided at any position suchas, for example, on the rear surface of the housing 50.

The terminal device 7 includes an extension connector 58 via whichanother device can be connected to the terminal device 7. The extensionconnector 58 is a communication terminal for exchanging data(information) with another device connected to the terminal device 7.The additional device connected to the extension connector 58 may be anydevice, and may be, for example, a game-specific controller (gun-shapedcontroller, etc.) or an input device such as a keyboard or the like. Theextension connector 58 may be omitted if there is no need to connect anadditional device to terminal device 7. The extension connector 58 mayinclude a terminal for supplying power to the additional device or aterminal for charging.

In addition to the extension connector 58, the terminal device 7includes a charging terminal 66 for obtaining power from an additionaldevice. When the charging terminal 66 is connected to a charging stand(not shown), power is supplied from the charging stand to the terminaldevice 7.

The terminal device 7 includes a charging connector, and the housing 50includes a cover portion 61 for protecting the charging connector. Thecharging connector can be connected to a charger 86 to be describedbelow, and power is supplied from the charger 86 to the terminal device7 when the charging connector is connected to the charger.

As shown in FIGS. 8( d) and 9, engagement holes 59 a and 59 b engageablewith tab portions of an additional device are provided in the bottomsurface of the projecting portion (the eaves portion 59). Engagementholes 50 a and 50 b are provided in the bottom surface of the housing50. In the case where the additional device is connected to the terminaldevice 7, four tab portions of the additional device are respectivelyengaged with the four engagement holes, which secures the terminaldevice 7 and the additional device with each other.

The terminal device 7 includes a battery cover 67 which can be attachedto and removed from the housing 50. A battery (battery 85 shown in FIG.10) is placed inside the battery cover 67. In the first embodiment, thebattery cover 67 is provided on the rear side of the housing 50, belowthe projecting portion (the eaves portion 59).

The housing 50 of the terminal device 7 includes holes 65 a and 65 bthrough which a strap cord can be tied to the terminal device 7. Theuser can tie a strap to one of the holes 65 a and 65 b, and fasten thestrap to the wrist of the user. Thus, even if the user drops theterminal device 7 or if the terminal device 7 comes off the hand, theterminal device 7 is prevented from falling or hitting another object.

With the terminal device 7 shown in FIGS. 8 and 9, the shape of eachoperation button, the shape of the housing 50, the number and thepositions of the components, etc., are merely examples, and the exampleembodiment can be realized with other shapes, numbers, and positions.

Now, an internal configuration of the terminal device 7 will bedescribed with reference to FIG. 10. FIG. 10 is a block diagram showingan internal configuration of the example terminal device 7. As shown inFIG. 10, in addition to the components shown in FIG. 8, the terminaldevice 7 includes a touch panel controller 71, a magnetic sensor 72, anacceleration sensor 73, the gyrosensor 74, a user interface controller(UI controller) 75, a codec LSI 76, the speakers 77, a sound IC 78, themicrophone 79, a wireless module 80, an antenna 81, the infraredcommunication module 82, a flash memory 83, a power supply IC 84, thebattery 85, and a vibrator 89. These electronic components are mountedon an electronic circuit board and accommodated in the housing 50.

The UI controller 75 is a circuit for controlling input/output of datato/from various types of input/output units. The UI controller 75 isconnected to the touch panel controller 71, the analog sticks 53 (analogsticks 53A and 53B), the operation buttons 54 (operation buttons 54Athrough 54L), the marker unit 55, the magnetic sensor 72, theacceleration sensor 73, the gyrosensor 74, and the vibrator 89. The UIcontroller 75 is connected to the codec LSI 76 and the extensionconnector 58. The power supply IC 84 is connected to the UI controller75, and power is supplied to various units via the UI controller 75. Thebuilt-in battery 85 is connected to the power supply IC 84 to supplypower. The charger 86 or a cable with which power can be obtained froman external power source can be connected to the power supply IC 84 viaa charging connector, and the terminal device 7 can be supplied withpower from an external power source, and also can be charged, using thecharger 86 or the cable. The terminal device 7 may be charged byattaching the terminal device 7 to a cradle (not shown) having acharging function.

The touch panel controller 71 is a circuit connected to the touch panel52 for controlling the touch panel 52. The touch panel controller 71generates input position data of a predetermined format based on asignal from the touch panel 52, and outputs the data to the UIcontroller 75. The input position data represents, for example,coordinates of a position on the input surface of the touch panel 52 atwhich an input has been made. The touch panel controller 71 reads asignal from the touch panel 52 and generates input position data at arate of once per predetermined amount of time. Various controlinstructions for the touch panel 52 are output from the UI controller 75to the touch panel controller 71.

The analog sticks 53 each output, to the UI controller 75, stick datarepresenting the direction and the amount of slide (or tilt) of thestick portion operated with a finger of the user. The operation buttons54 each output, to the UI controller 75, operation button datarepresenting an input state of each of the operation buttons 54A through54L (e.g., whether the button has been pressed or not).

The magnetic sensor 72 detects an azimuthal direction by sensing thesize and direction of the magnetic field. Azimuthal direction datarepresenting the detected azimuthal direction is output to the UIcontroller 75. A control instruction for the magnetic sensor 72 isoutput from the UI controller 75 to the magnetic sensor 72. While thereare sensors using an MI (magnetic impedance) element, a fluxgate sensor,a Hall element, a GMR (giant magneto-resistive) element, a TMR (tunnelmagneto-resistance) element, an AMR (anisotropic magneto-resistive)element, etc., the magnetic sensor 72 may be any sensor as long as it ispossible to detect the azimuthal direction. Strictly describing, in aplace where there is a magnetic field other than a geomagnetic field,the obtained azimuthal direction data does not represent the azimuthaldirection. Nevertheless, when the terminal device 7 moves, the azimuthaldirection data changes even in such a case. Therefore, it possible tocalculate the change in the attitude of the terminal device 7.

The acceleration sensor 73 is provided inside the housing 50 fordetecting the magnitude of a linear acceleration along each ofdirections of three axes (x, y and z axes shown in FIG. 8( a)).Specifically, the acceleration sensor 73 detects the magnitude of thelinear acceleration along each of the axes, where the x axis lies in alongitudinal direction of the housing 50, the y axis lies in a directionvertical to the front surface of the housing 50, and the z axis lies ina width direction of the housing 50. Acceleration data representing eachdetected acceleration is output to the UI controller 75. A controlinstruction for the acceleration sensor 73 is output from the UIcontroller 75 to the acceleration sensor 73. While the accelerationsensor 73 is assumed to be, for example, a capacitive-type MEMS-typeacceleration sensor in the first embodiment, other types of accelerationsensors may be employed in other embodiments. The acceleration sensor 73may be an acceleration sensor for detecting an acceleration along oneaxis or two axes.

The gyrosensor 74 is provided inside the housing 50 for detectingangular velocities about the three axes, i.e., the x-axis, the y-axisand the z-axis. Angular velocity data representing each detected angularvelocity is output to the UI controller 75. A control instruction forthe gyrosensor 74 is output from the UI controller 75 to the gyrosensor74. The number and combination of gyrosensors used for detecting angularvelocities about the three axes may be any number and combination, andthe gyrosensor 74 may be formed by a 2-axis gyrosensor and a 1-axisgyrosensor, as is the gyrosensor 48. The gyrosensor 74 may be agyrosensor for detecting an angular velocity about one axis or angularvelocities about two axes.

The vibrator 89 is, for example, a vibration motor or a solenoid, and isconnected to the UI controller 75. The terminal device 7 is vibrated byactuation of the vibrator 89 in response to a command from the UIcontroller 75. Thus, a so-called vibration-feedback game is realized, inwhich the vibration is conveyed to the user's hand holding the terminaldevice 7.

The UI controller 75 outputs, to the codec LSI 76, operation dataincluding input position data, stick data, operation button data,azimuthal direction data, acceleration data, and angular velocity datareceived from various components described above. In the case whereanother device is connected to the terminal device 7 via the extensionconnector 58, data representing an operation performed on the anotherdevice may be further included in the operation data.

The codec LSI 76 is a circuit for performing a compression process ondata to be transmitted to the game device 3, and a decompression processon data transmitted from the game device 3. The LCD 51, the camera 56,the sound IC 78, the wireless module 80, the flash memory 83, and theinfrared communication module 82 are connected to the codec LSI 76. Thecodec LSI 76 includes a CPU 87 and an internal memory 88. While theterminal device 7 does not perform a game process itself, the terminaldevice 7 may execute a minimal possible program for the managementthereof and for the communication. A program stored on the flash memory83 is read out to the internal memory 88 and executed by the CPU 87 uponpower-up, and thus the terminal device 7 is started. A part of an areaof the internal memory 88 is used as the VRAM for the LCD 51.

The camera 56 captures an image in response to an instruction from thegame device 3, and outputs the captured image data to the codec LSI 76.A control instructions for the camera 56, such as an image-capturinginstruction or the like, is output from the codec LSI 76 to the camera56. The camera 56 can also capture a moving image. That is, the camera56 can repeatedly capture images and repeatedly output the image data tothe codec LSI 76.

The sound IC 78 is a circuit connected to the speakers 77 and themicrophone 79 for controlling input/output of sound data to/from thespeakers 77 and the microphone 79. That is, when receiving sound datafrom the codec LSI 76, the sound IC 78 outputs a sound signal obtainedby performing D/A conversion on the sound data to the speakers 77 sothat sound is output from the speakers 77. The microphone 79 detects asound propagated to the terminal device 7 (voice of the user, etc.), andoutputs a sound signal representing such a sound to the sound IC 78. Thesound IC 78 performs A/D conversion on the sound signal from themicrophone 79 to output sound data of a predetermined format to thecodec LSI 76.

The infrared communication module 82 emits an infrared signal to performinfrared communication with another device. Here, the infraredcommunication module 82 has, for example, a function of performinginfrared communication in accordance with the IrDA standard and afunction of outputting an infrared signal (control signal) forcontrolling the TV 2.

The codec LSI 76 transmits, as terminal operation data, image data fromthe camera 56, sound data from the microphone 79, and operation datafrom the UI controller 75 to the game device 3 via the wireless module80. In the first embodiment, the codec LSI 76 performs a compressionprocess similar to that of the codec LSI 27 on the image data and thesound data. The terminal operation data and the compressed image dataand sound data are output, as transmission data, to the wireless module80. The antenna 81 is connected to the wireless module 80, and thewireless module 80 transmits the transmission data to the game device 3via the antenna 81. The wireless module 80 has a similar function tothat of the terminal communication module 28 of the game device 3. Thatis, the wireless module 80 has a function of connecting to a wirelessLAN by a scheme in conformity with the IEEE 802.11n standard, forexample. The transmitted data may be encrypted when necessary or may notbe encrypted.

As described above, the transmission data transmitted from the terminaldevice 7 to the game device 3 includes operation data (terminaloperation data), image data, and sound data. In the case where anotherdevice is connected to the terminal device 7 via the extension connector58, data received from the another device may be further included in thetransmission data. The codec LSI 76 may transmit, to the game device 3,data received via infrared communication by the infrared communicationmodule 82 as being included in the transmission data when necessary.

As described above, compressed image data and sound data are transmittedfrom the game device 3 to the terminal device 7. These types of data arereceived by the antenna 81 (reception unit) and transferred to the codecLSI 76 via the wireless module 80. The codec LSI 76 decompresses thereceived image data and sound data. The decompressed image data isoutput to the LCD 51, and the image is displayed on the LCD 51. That is,the codec LSI 76 (the CPU 87) causes the received image data to bedisplayed on the display unit. The decompressed sound data is output tothe sound IC 78, and the sound IC 78 causes sound to be output from thespeakers 77.

In the case where terminal control data is included in data receivedfrom the game device 3, the codec LSI 76 and the UI controller 75 eachissue a control instruction to various units in accordance with theterminal control data. As described above, the terminal control data isdata representing a control instruction for each of the components ofthe terminal device 7 (in the first embodiment, the camera 56, the touchpanel controller 71, the marker unit 55, the sensors 72 through 74, andthe infrared communication module 82). In the first embodiment, thecontrol instruction represented by the terminal control data may be aninstruction to activate the operation of each of the components ordeactivate (stop) the operation thereof. That is, components that arenot used in a game may be deactivated in order to reduce the powerconsumption, in which case it is ensured that data from the deactivatedcomponents are not included in the transmission data to be transmittedfrom the terminal device 7 to the game device 3. For the marker unit 55,which includes infrared LEDs, the control can be done simply by turningON/OFF the power supply thereto.

The game device 3 can control an output of the infrared communicationmodule 82, and thus controls the operation of the TV 2. Specifically,the game device 3 outputs, to the terminal device 7, an instruction(terminal control data described above) to cause the infraredcommunication module 82 to output an infrared signal corresponding to acontrol command for controlling the TV 2. In response to thisinstruction, the codec LSI 76 causes the infrared communication module82 to output an infrared signal corresponding to the control command.Here, the TV 2 includes an infrared reception unit capable of receivingan infrared signal. By the infrared reception unit receiving theinfrared signal output by the infrared communication module 82, the TV 2operates in accordance with the infrared signal. The instruction fromthe game device 3 may indicate an infrared signal itself, or when theterminal device 7 has an infrared signal pattern stored thereon, mayindicate the pattern.

As described above, while the terminal device 7 includes the operatingunits such as the touch panel 52, the analog sticks 53 and the operationbuttons 54. In other embodiments, other operating units may be includedinstead of, or in addition to, these operating units.

The terminal device 7 includes the magnetic sensor 72, the accelerationsensor 73 and the gyrosensor 74 as the sensors for calculating themovement of the terminal device 7 (including the position and theattitude thereof, or changes in the position and the attitude thereof).In other embodiments, the terminal device 7 may include only one or twoof these sensors, or may not include any of these sensors. In stillother embodiments, other sensors may be included instead of, or inaddition to, these sensors.

While the terminal device 7 includes the camera 56 and the microphone79, the terminal device 7 may not include the camera 56 or themicrophone 79, or may include only one of them in other embodiments.

While the terminal device 7 includes the marker unit 55 as a componentfor calculating the positional relationship between the terminal device7 and the controller 5 (the position and/or attitude, etc., of theterminal device 7 as seen from the controller 5), the terminal device 7may not include the marker unit 55 in other embodiments. In still otherembodiments, the terminal device 7 may include other mechanisms as acomponent for calculating the positional relationship. For example, inother embodiments, the controller 5 may include a marker unit, and theterminal device 7 may include an image-capturing element. In such acase, the marker device 6 may include an image-capturing element,instead of an infrared LED.

1-7. Operation of Video Display System

Now, an operation of the video display system 90 will be described.Hereinafter, a case will be described in which the game system 1requests the transmission server 91 to acquire the video information(selection instruction), the transmission server 91 transmits the videoinformation in response to the request, and the game system 1 receivesthe video information and displays the video content and theadvertisement information on the two display devices. In an exampledescribed below, a TV program which has been already broadcast (programwhich the user has missed) is provided as the video content, and theproducer of the TV program (broadcaster) uses the transmission server 91to provide the TV program to the game system 1.

First, various types of data usable for the operation of the videodisplay system 90 will be described. FIG. 11 shows an example of datastored on a storage unit of the game device 3 (including the externalmain memory 12, the internal main memory 11 e, and the flash memory 17).As shown in FIG. 11, a control program 110, operation data 111, videoinformation data 112, and processing data 116 are stored on the storageunit of the game device 3. In addition to the data shown in FIG. 11,various types of data usable for a process executed by the game device 3(image data, sound data and the like) are stored on the storage unit.The data shown in FIG. 11 may be stored on any other storage deviceaccessible to the game device 3, as well as on the storage unit in thegame device 3.

The control program 110 is a program for causing a CPU 10 to execute aprocess in the game device 3 (game system 1). In the first embodiment,the CPU 10 executes the control program 110, and thus steps shown in theflowchart shown in FIG. 12 are executed. The control program 110 ispartially or entirely read from the flash memory 17 at an appropriatetiming after the game device 3 is powered on, and thus is stored on thestorage unit. The control program 110 may be acquired from the opticaldisc 4 or another device external to the game device 3 (e.g., via theInternet), instead of from the flash memory 17.

The operation data 111 is data representing an operation performed by aplayer on the terminal device 7 (terminal operation data). The operationdata 111 is transmitted from the terminal device 7, acquired by the gamedevice 3, and stored on the storage unit. The game device 3 iscommunicable with a plurality of terminal devices, and can acquireoperation data from each of the plurality of terminal devices. In thecase where there are a plurality of terminal devices, terminal operationdata transmitted from each of the terminal devices is stored on thestorage unit. On the storage unit, a predetermined number of pieces ofterminal operation data may be stored sequentially from the latest data(the last data acquired) for each terminal device.

In the first embodiment, the operation data 111 includes angularvelocity data, acceleration data, operation button data, stick data,touch position data, and azimuthal direction data. The angular velocitydata is data representing an angular velocity detected by the gyrosensor74. The acceleration data is data representing an acceleration(acceleration vector) detected by the acceleration sensor 73. Theoperation button data is data representing an input state on each of theoperation buttons 54A through 54L provided in the terminal device 7.Specifically, the operation button data represents whether each of theoperation buttons 54A through 54L has been pressed or not. Stick data isdata representing a direction in which and a magnitude by which thestick portion of each of the analog sticks 53 (analog sticks 53A and53B) has been slid (or tilted). The touch position data is datarepresenting a position on the input surface of the touch panel 52 atwhich an input has been made (touch position). In the first embodiment,the touch position data represents coordinate values of atwo-dimensional coordinate system which indicate the position on theinput surface. In the case where the touch panel 52 is of themulti-touch type, the touch position data may represent a plurality oftouch positions. The azimuthal direction data is data representing anazimuthal direction detected by the magnetic sensor 72.

The operation data 111 may include either one of the above-mentionedtypes of data included in the operation data 111 in the firstembodiment. In the case where the terminal device 7 includes anotherinput mechanism (e.g., touch pad, image-capturing mechanism of thecontroller 5, or the like), the operation data 111 may include datarepresenting an operation on the another input mechanism.

The video information data 112 is data representing the videoinformation acquired by the game device 3. In the first embodiment, thevideo information data 112 includes content data 113, advertisement data114, and reproduction control data 115. The content data 113 is datarepresenting the video content included in the video information. Theadvertisement data 114 is data representing the advertisementinformation included in the video information. The reproduction controldata 115 is data representing an instruction to control reproduction ofthe advertisement information by the terminal device 7 (reproductioncontrol instruction).

The processing data 116 is data usable in a video display process (FIG.12) described later. The processing data 116 includes transmission data117 and control command data 118. In addition to the data shown in FIG.11, the processing data 116 includes various types of data usable in thecontrol program 110.

The transmission data 117 represents transmission information to betransmitted to the transmission server 91 from the game system 1 (gamedevice 3). The transmission data 117 represents attribute information onthe user and/or current information. The attribute information isinformation regarding the user of the game system 1, for example, theage, address (district in which the user resides), the family structureor the like of the user. The current information represents a currentstate (e.g., current time, current weather, etc.). In the firstembodiment, the transmission data 117 includes both of the datarepresenting the attribute information and the data representing thecurrent information.

The control command data 118 is data representing a control command forcontrolling the TV 2. In the first embodiment, data representing varioustypes of control commands for causing the TV 2 to perform various typesof operations is stored on the storage unit in the game device 3 inadvance. The control command data 118 represents the control command tobe transmitted to the TV 2 among these various types of controlcommands.

Now, with reference to FIGS. 12 through 17, processes executed by thegame device 3 in the first embodiment will be described in detail. FIG.12 is a main flowchart showing an example of process flow executed bythe game device 3 in the first embodiment. When the game device 3 ispowered on, the CPU 10 of the game device 3 executes a boot programstored on a boot ROM (not shown) to initialize units including the mainmemory (external main memory 12 or internal main memory 11 e). Then, thecontrol program 110 stored on the flash memory 17 is read into the mainmemory, and the CPU 10 starts executing the control program 110. Theflowchart shown in FIG. 12 shows the process executed after theabove-described process is completed. The game device 3 may beconfigured to execute the control program 110 soon after being poweredon. Alternatively, the game device 3 may be configured as follows. Afterthe game device 3 is powered on, a built-in program for displaying apredetermined menu screen is executed, and then it is instructed to bootthe control program 110 by, for example, a selection operation on themenu screen. In response to this, the control program 110 is executed.

The steps shown in each of the flowcharts shown in the figures aremerely examples, and the processing order of the steps may be exchangedas long as substantially the same results are provided (same as in asecond embodiment). The values of the variables and the threshold valuesused in determination steps are also merely examples, and other valuesmay be adopted when necessary. In this specification, each of the stepsin the flowcharts will be described as being executed by the CPU 10.Alternatively, the steps in the flowcharts may be partially executed bya processor other than the CPU 10 or a dedicated circuit.

First, in step S1, the CPU 10 accesses the transmission server 91.Specifically, the CPU 10 requests the transmission server 91 for aselection image usable for selecting a video content, and acquires anddisplays the selection image. The selection image is, for example, a webpage which introduces video contents. In the case where there are aplurality of accessible servers (e.g., in the case where a transmissionserver is installed for each of a plurality of broadcasters), the CPU 10may specify the transmission server to access in accordance with apredetermined condition. For example, the transmission server may bespecified in accordance with an instruction of the user or based on theattribute information on the user described later. Any specific methodfor specifying the transmission server is usable. For example, the CPU10 may access a server which provides a web page linked to thetransmission servers so that the user can specify the transmissionserver on the web page.

In step S1, the CPU 10 transmits an acquisition request for acquiringthe selection image to the transmission server 91. Specifically, the CPU10 stores data on the acquisition request in a predetermined area of theflash memory 17 as data to be transmitted to the network 92. Theinput/output processor 11 a transmits the data on the acquisitionrequest stored on the flash memory 17 to the network 92 at apredetermined timing. Owing to this, the data on the acquisition requestis transmitted to the transmission server 91. When the acquisitionrequest is received by the transmission server 91, the selection imageis transmitted from the transmission server 91 to the game device 3.Hereinafter, with reference to FIG. 13, the process executed by thetransmission server 91 will be described.

FIG. 13 is a main flowchart showing an example of process flow executedby the transmission server 91 in the first embodiment. A series ofprocesses shown in FIG. 13 is executed by the processing unit 12 of thetransmission server 91 for executing a predetermined control program.The processes shown in FIG. 13 is started in response to an access madeto the transmission server 91 by the game system 1. When the access fromthe game system 1 is discontinued, the processing unit 102 terminatesthe processes shown in FIG. 13. In the case where a plurality of gamesystems access the transmission server 91, the processing unit 102executes the process shown in FIG. 13 in parallel for the plurality ofgame systems.

First, in step S11, the processing unit 102 transmits the selectionimage in response to the acquisition request from the game device 3. Atthis point, the data on the web page of the selection image has beenstored on a predetermined storage unit of the transmission server 91.The processing unit 102 reads the data and transmits the data to thegame device 3 via the communication unit 101. After the process of stepS11, the transmission server 91 executes step 12 described later.

As a result of the process of step S11, the game device 3 receives theselection image. The CPU 10 outputs the received selection image to atleast one of the two display devices (TV 2 and terminal device 7) anddisplays the selection image. In step S1, the image displayed on theside of the game system 1 (selection image, etc.) may be displayed onthe TV 2 or on the terminal device 7. In the case where the image isdisplayed on the terminal device 7, the user can operate the imageeasily by use of the touch panel 52 or the like and also can use the TV2 for another purpose, which is convenient. The operation by the user instep S1 may be performed in any manner. In the first embodiment, theoperation by the user in step S1 is performed by use of the terminaldevice 7. After the process of step S1 described above, the process ofstep S2 is executed.

In step S2 shown in FIG. 12, the CPU 10 specifies the video content tobe reproduced. Specifically, the CPU 10 accepts, from the user, aselection operation of selecting one of the video contents included inthe selection image and specifies the video content selected by theselection operation. The selection operation may be performed in anymanner. In the first embodiment, the selection operation is performed byuse of the terminal device 7. For example, a selection image includingan image representing each video content is displayed on the terminaldevice 7, and in response to the operation of touching the imagerepresenting a certain video content, the certain video content isspecified. When the video content is specified, the CPU 10 stores dataon the information identifying the specified video content (referred toas “identified information”, for example, ID attached to the videocontent) on the main memory. After the process of step S3, the processof step S3 is executed.

In step S3, the CPU 10 acquires the transmission information to betransmitted to the transmission server 91. In the first embodiment,attribute information representing the attribute of the user and currentinformation representing the current state are acquired as thetransmission information. The transmission information may be acquiredin any manner. For example, the CPU 10 may acquire information stored onthe game device 3 (game system 1) in advance. In the case where, forexample, attributer information on the age, the district of residence,the family structure or the like of the user is registered in the gamesystem 1 in advance, such attribute information may be acquired as thetransmission information. In the case where attribute information on aplurality of users is registered, the CPU 10 may identify the user whois currently using the terminal device 7 based on, for example, an imageof the user captured by the camera 56 of the terminal device 7 toacquire the attribute information on the user. The CPU 10 may acquiretransmission information on the current information on the currentweather or the like from an external device via the network 92. Datarepresenting the transmission information acquired as described above isstored on the main memory as the transmission data 117. After theprocess of step S3, the process of step S4 is executed.

In step S4, the CPU 10 issues a selection instruction on the videoinformation (acquisition request) to the transmission server 91. Theselection instruction includes the above-mentioned identifyinginformation and transmission information. Specifically, the CPU 10 readsdata on the identifying information and the transmission data 117 fromthe main memory, and generates selection instruction data including theread data. The selection instruction data is transmitted to thetransmission server 91 in substantially the same manner as theacquisition request in step S1. After the process of step S4, the gamedevice 3 executes the process of step S5.

In response to the selection instruction on the video information beingtransmitted to the transmission server 91 by the process of step S4executed by the game system 1, the video information is transmitted fromthe transmission server 91. Specifically, in step S12 after step S11,the processing unit 102 of the transmission server 91 determines whetherthe selection instruction on the video information has been received ornot. When the determination result of step S12 is positive, the processof step S13 is executed. By contrast, when the determination result ofstep S12 is negative, the process of step S12 is executed again. Namely,the processing unit 102 waits until the selection instruction isreceived from the game device 3, and upon receipt thereof, executes theprocess of step S13.

In step S13, based on the selection information on the videoinformation, the processing unit 102 selects the video content to betransmitted (video content to be displayed on the TV 2). Specifically,among video contents stored on the video storage unit 103, the videocontent which is identified by the ID included in the selectioninstruction is selected as the video content to be transmitted. Afterthe process of step S13, the process of step S14 is executed.

In step S14, based on the transmission information included in theselection instruction on the video information, the processing unit 102selects the advertisement information to be transmitted. Theadvertisement information selected in this step is the advertisementinformation to be displayed on the terminal device 7 when the videocontent selected in step S13 is displayed on the TV 2. Specifically,among advertisement information stored on the advertisement storage unit104, the advertisement information corresponding to the transmissioninformation is selected as the advertisement information to betransmitted. The selection of the advertisement information based on thetransmission information may be performed in any specific manner. Forexample, the transmission server 91 prepares, in advance, correspondenceinformation which represents the correspondence between a conditionregarding a predetermined item included in the transmission information(e.g., the condition that the age of the user is between 30 and 40, orthe condition that the district where the user resides is in apredetermined district) and advertisement information which is to beselected when such a condition is fulfilled. In step S14, the CPU 10uses the correspondence information to select the advertisementinformation corresponding to the transmission information. In the casewhere the reproduction time duration of one piece of advertisementinformation is shorter than the reproduction time duration of the videocontent selected in step 13, a plurality of pieces of advertisementinformation may be selected. The advertisement information to beselected may be changed in accordance with the video content. Namely,information which represents the correspondence between video contentsand advertisement information which is to be reproduced when each of thevideo contents is reproduced may be prepared; and in step S14, the CPU10 may select the advertisement information to be transmitted among theadvertisement information corresponding to the video content selected instep S13. After the process of step S14, the process of step S15 isexecuted.

As described above, in the first embodiment, the game device 3 acquiresthe transmission information representing the attribute of the userand/or the current state (step S3), and transmits the transmissioninformation to the transmission server 91 (step S4). Based on thetransmission information, the transmission server 91 determines theadvertisement information to be transmitted to the game device 3, whichis an addressee of the received transmission information (step S14).According to the first embodiment, the transmission server 91 canpresent the viewer with the advertisement information corresponding tothe attribute of the viewer of the video content or the current state.Owing to this, appropriate advertisement information corresponding tothe age, the district of residence or the like of the viewer ispresented, or appropriate advertisement information corresponding to thetime or the like when the video content is to be viewed is presented.Thus, the advertizing effect can be improved.

In step S15, the processing unit 102 transmits the video information tothe game device 3. The video information includes the video content andthe advertisement information selected in steps S13 and S14. The videoinformation may be transmitted in any manner. In the first embodiment,the processing unit 102 performs streaming transmission of the videoinformation. Specifically, the processing unit 102 reads the videocontents stored on the video storage unit 103 and the advertisementinformation stored on the advertisement storage unit 104, and transmitsthe moving images of the video contents and the advertisementinformation in succession. In the case where, for example, theadvertisement information does not need to be transmitted for the sametime duration as that of the video content (e.g., in the case where theadvertisement information is reproduced by the terminal device 7 inrepetition), only the video content may be transmitted temporarily. Inother embodiments, the game device 3 may first download the entirety ofthe video information and then reproduce the video information. Thetransmission server 91 may process, for example, encrypt the videoinformation, and the video information may be decoded on the side of thegame system 1. In this case, the video content and the advertisementinformation may be encrypted separately, or the video informationincluding the video content and the advertisement information may beencrypted.

As described above, in the first embodiment, the transmission server 91transmits the video information including a combination of the videocontent and the advertisement information to the game system 1. Owing tothis, the video content and the advertisement information which is to bepresented together with the video content can be transmitted as a set.This allows the side of the transmission server (broadcaster) to adjustthe content of advertisement easily.

In the first embodiment, the transmission server 91 performs streamingtransmission of the video content and the advertisement informationincluding a moving image to the game device 3. In this case, the gamedevice 3 outputs the video content received in succession to the TV 2 insuccession while transmitting the advertisement information received insuccession to the terminal device 7 in succession (steps S6 and S7described later). Owing to this, predetermined advertisement informationcan be reproduced at an appropriate timing in accordance with a specificcontent of the video content. For example, predetermined advertisementinformation is reproduced at a timing when the video content provides apredetermined specific content (e.g., in the case where the videocontent is a relay broadcasting program of a baseball game, at a timingwhen the offense and defense are exchanged), and thus the advertizingeffect of the advertisement information can be improved. According tothe first embodiment, advertisement information suitable to the videocontent can be transmitted from the side of the server 91 easily.Therefore, advertisement information suitable to the video content canbe presented to the user (viewer), and thus the advertizing effect canbe improved.

In the first embodiment, the video information may include areproduction control instruction. The reproduction control instructionis an instruction to control the reproduction of the advertisementinformation by the terminal device 7. For example, the reproductioncontrol instruction may be an instruction to reproduce the advertisementinformation, which is a target of control, in repetition. Alternatively,the reproduction control instruction may be an instruction to reproducethe advertisement information, which is a target of control, in responseto the reproduced specific content of the video content reaching apredetermined state (e.g., in response to a predetermined specificcontent having been reproduced, or in response to a predetermined timeduration having passed from the start of the reproduction). Thereproduction control instruction may be issued in the form of datarepresenting a condition for reproduction or in the form of a programexecutable by the game device 3. In step S15, the CPU 10 transmits thevideo information in the state of including the reproduction controlinstruction at an appropriate timing during the streaming transmissionof the video information. For example, the reproduction controlinstruction may be transmitted before the advertisement information,which is a target of control, is transmitted.

The transmission of the video information in step S15 is continued untilthe entirety of the video information is transmitted. When thetransmission of the video information is completed, the process of stepS12 is executed again. After this, the processing unit 102 executes theprocesses of steps S12 through S15 in repetition until the access to thetransmission server 91 by the game system 1 is discontinued.

Returning to FIG. 12, in step S5, the CPU 10 executes the reproductioncontrol process. The reproduction control process is a process ofcausing the video content and the advertisement information included inthe video information from the transmission server 91 to be displayed onthe two display devices. In the first embodiment, the reproductioncontrol process is executed in repetition at a rate of once in apredetermined time duration (1 frame time). Hereinafter, with referenceto FIG. 14, the reproduction control process will be described indetail.

FIG. 14 is a flowchart showing an example of process flow of thereproduction control process shown in FIG. 12 (step S5) in detail. Thereproduction control process is executed as follows. First, in step S21,the CPU 10 acquires the operation data (terminal operation data)transmitted from the terminal device 7. The terminal device 7 transmitsthe terminal operation data to the game device 3 in repetition, and thusthe game device 3 receives the terminal device operation in succession.In the game device 3, the terminal communication module 28 receives theterminal operation data in succession, and the input/output processor 11a stores the terminal operation data on the main memory in succession.In step S21, the CPU 10 reads the latest terminal operation data 111from the main memory. After the process of step S21, the process of stepS22 is executed.

In step S22, the CPU 10 receives (acquires) the video informationtransmitted from the transmission server 91. Namely, the data receivedfrom the transmission server 91 is stored on the flash memory 17.Specifically, the data representing the video content included in thevideo information is stored on the flash memory 17 as the content data113, the data representing the advertisement information included in thevideo information is stored on the flash memory 17 as the advertisementdata 114, and the data representing the reproduction control instructionis stored on the flash memory 17 as the reproduction control data 115.Since the transmission server 91 transmits the video information insuccession by streaming transmission, each type of the data included inthe video information is stored on the flash memory 17 in succession.Among the data stored on the flash memory 117, data on the image (movingimage or still image) which has been reproduced may be erased whenappropriate. After the process of step S22, the process of step S23 isexecuted.

In step S23, the CPU 10 determines whether an operation regarding thereproduction of the video content (reproduction operation) has beenperformed or not. The reproduction operation may be any operation whichinstructs to control the reproduction of the video content. In the firstembodiment, the reproduction operation is an operation of reproducingthe video content, stopping the reproduction, or temporarily pausing thereproduction. In other embodiments, the reproduction operation may be anoperation of, for example, fast-forwarding or reversing the reproductionof the video content, or an operation of, for example, adjusting thevolume of the sound which is output together with the video content. Thereproduction operation may be performed in any manner. In the firstembodiment, the reproduction operation is performed by use of theterminal device 7. In the first embodiment, the user can perform thereproduction operation by use of the touch panel 52, which will bedescribed later in detail regarding step S29. In step S23, the processis executed as follows specifically. Based on the operation data read instep S21, the CPU 10 determines whether the reproduction operation hasbeen performed or not. When the determination result of step S23 isnegative, the process of step S24 is executed. By contrast, when thedetermination result of step S23 is positive, the process of step S25 isexecuted.

In step S24, the CPU 10 controls the reproduction of the video content.Specifically, the CPU 10 specifies an image of a video content to beoutput to the TV 2 among the images of the video contents stored on theflash memory 17. In step S24, the video contents are specified in a timeseries such that normal reproduction is performed. After the process ofstep S24, the process of step S26 described later is executed.

In step S25, the CPU 10 controls the reproduction of the video contentin accordance with the reproduction operation. Specifically, the CPU 10specifies an image among the images of the video contents stored on theflash memory 17 such that the image of the video content to be output tothe TV 2 is suitable to the reproduction operation (fast-forwarding,reversing, pausing, etc.). After the process of step S25, the process ofstep S26 is executed.

In the first embodiment, the user can perform an operation on the TV 2by use of the terminal device 7. Specifically, the TV 2 receives acontrol signal, generated in accordance with an operation performed bythe user on the terminal device 7, from the terminal device 7 or thegame device 3, and performs an operation in accordance with the controlsignal. Hereinafter, the processes of steps S26 and S27 for controllingthe operation of the TV 2 will be described in detail.

In step S26, the CPU 10 determines whether an operation for controllingthe TV 2 (TV operation) has been performed or not. The TV operation maybe any operation which instructs to control the operation of the TV 2.In the first embodiment, the TV operation is an operation of, forexample, turning ON/OFF the power of the TV 2, increasing or decreasingthe sound volume, or changing the display mode. In other embodiments, inthe case where the TV 2 can control the reproduction of the videocontent, the TV operation may be an operation (on the TV 2) regardingthe reproduction of the video content, for example, an operation ofreproducing the video content, or fast-forwarding or reversing thereproduction. The TV operation may be performed in any manner. In thefirst embodiment, the TV operation is performed by use of the terminaldevice 7. In the first embodiment, the user can perform the TV operationby use of the touch panel 52, like the reproduction operation, whichwill be described later in detail. In step S26, the process is executedas follows specifically. Based on the operation data read in step S21,the CPU 10 determines whether the TV operation has been performed ornot. When the determination result of step S26 is positive, the processof step S27 is executed. By contrast, when the determination result ofstep S26 is negative, the process of step S27 is skipped and the processof step S28 is executed.

In step S27, the CPU 10 controls the TV 2 in accordance with the TVoperation performed by the user. Specifically, a control command forcausing the TV 2 to perform an operation in accordance with the TVoperation is generated and is output to the TV 2. The CPU 10 stores thecontrol command data 118 representing the control command on the storageunit. The TV 2 may be controlled by the control command in any manner.In the game system 1 in the first embodiment, the TV 2 can be controlledby a first method and/or a second method described below.

According to the first method, the terminal device 7 transmits a controlsignal regarding the reproduction and/or the display of the videocontent to the TV 2 in response to an operation performed by the user onthe terminal device 7, and the TV 2 performs an operation in accordancewith the control signal from the terminal device 7. In the firstembodiment, the CPU 10 transmits, to the terminal device 7, aninstruction to cause the infrared communication module 82 to output aninfrared signal corresponding to the control command represented by thecontrol command data 118. In response to this instruction, the codec LSI76 of the terminal device 7 outputs the infrared signal corresponding tothe control command to be output to the infrared communication module82. As a result of the infrared signal being received by the infraredreception unit of the TV 2, the TV 2 performs an operation in accordancewith the control command. In the first embodiment, the terminal device 7controls the TV 2 by infrared communication using an infrared receptionfunction owned by a general TV. Therefore, a general TV can be usedeasily as the display device, and thus a system usable for a variety ofapplications can be provided.

According to the second method, the game device 3 outputs the controlsignal to the TV 2, and the TV 2 performs an operation in accordancewith the control signal. Specifically, in the first embodiment, the CPU10 outputs the control signal corresponding to the control commandrepresented by the control command data 118 to the TV 2 via the AVconnector 16. The game system 1 controls the operation of the TV 2 by atleast either the first method or the second method. After the process ofstep S27, the process of step S28 is executed.

As described above, in the first embodiment, the terminal device 7includes an operating unit for accepting an operation performed by theuser, and transmits the operation data representing an operationperformed on the operating unit to the game device 3. The game device 3acquires the operation data (step S21), and performs control regardingthe reproduction and/or the display of the video content on the TV 2based on the operation data (steps S25 and S28). In the firstembodiment, the “control regarding the reproduction of the videocontent” is the control in the process of step S25. In the firstembodiment, the “control regarding the display of the video content” isthe control of turning ON/OFF the TV 2 in the process of step S28. Asdescribed above, in the first embodiment, the user can perform anoperation on the video content by use of the terminal device 7.According to the first embodiment, the user views the terminal device 7when performing an operation on the video content. At this point, it ispossible to direct the user's attention to also the advertisementinformation displayed on the terminal device 7. Namely, in the firstembodiment, an operation on the video content is allowed to be performedby use of the terminal device 7 (which displays the advertisementinformation), and thus it is possible to direct the user's attentionalso to the advertisement information displayed on the terminal device 7during the reproduction of the video content. This can improve theadvertizing effect of the advertisement information.

In step S28, the CPU 10 performs reproduction control on theadvertisement information. Specifically, the CPU 10 specifies an imageto be output to the terminal device 7 among images of the advertisementinformation stored on the flash memory 17. The specification of theimage in step S28 may be performed in any manner. In the firstembodiment, in the case where there is a reproduction controlinstruction from the transmission server 91, the image is specified inaccordance with the reproduction control instruction. By contrast, inthe case where is no reproduction control instruction from thetransmission server 91, the images are specified in a time series suchthat normal reproduction is performed. Specifically, the CPU 10determines whether the reproduction control data 115 is stored on theflash memory 17 or not. When the reproduction control data 115 isstored, the CPU 10 reads the reproduction control data 115 and specifiesthe image such that the advertisement information is reproduced inaccordance with the reproduction control instruction. By contrast, whenthe reproduction control data 115 is not stored, the CPU 10 specifiesthe images such that the images are reproduced in a time series.

As described above, in the first embodiment, the transmission server 91transmits the video information including the reproduction controlinstruction regarding the reproduction of the advertisement information(step S15), and the CPU 10 controls the reproduction of theadvertisement information in accordance with the reproduction controlinstruction (step S28). Owing to this, the reproduction of theadvertisement information by the terminal device 7 can be controlled onthe side of the transmission server 91 easily. For example, theadvertisement information can be reproduced in a form described below.

FIG. 15 shows an example of content to be reproduced by the TV 2 and theterminal device 7. In the first embodiment, as shown in, for example,FIG. 15, the CPU 10 may control a predetermined advertisement A to bereproduced in repetition in accordance with the reproduction controlinstruction (in this case, the reproduction time duration of thepredetermined advertisement A is shorter than that of the video contentto be reproduced by the TV 2). The advertisement A is, for example, acommercial provided as one piece of advertisement information. Owing tothis, the moving image of the advertisement information is reproduced inrepetition, and thus the advertizing effect can be improved. Inaddition, the use of the reproduction control instruction can reduce theamount of data of the video information to be transmitted from thetransmission server 91.

The CPU 10 may control the reproduction of the advertisement informationby the terminal device 7 such that predetermined advertisementinformation is reproduced by the terminal device 7 at a timing when thespecific content of the video content reaches a predetermined state.Specifically, as shown in FIG. 15, the CPU 10 may control thereproduction such that a predetermined advertisement B is reproduced bythe terminal device 7 at a timing T1 when the video content reproducedby the TV 2 reaches a predetermined state.

The predetermined state may be any state, and the timing may be anytiming. The timing T1 is, for example, a timing when the user is assumedto terminate the viewing (attention) of the video content, or a timingwhen the user's attention is assumed to be diverged from the videocontent. Specifically, in the case where the video content is a relaybroadcasting program of a baseball game, the timing T1 may be a timingwhen the offense and defense are exchanged. In the case where the videocontent include a commercial, the timing T1 may be a timing when thevideo content is switched from the TV program to the commercial. In thismanner, the predetermined advertisement B is reproduced by the terminaldevice 7 at a timing when the attention of the user is assumed to bediverged from the video content displayed on the TV 2. Owing to this, itis possible to cause the user to notice the advertisement B easily,which can improve the advertizing effect of the advertisement B.

The advertisement information may be related to the specific content ofthe video content. In this case, the timing T1 may be, for example, atiming when the content related to the advertisement B is reproduced asthe video content. Specifically, in the case where the video contentincludes advertizing moving images (commercials), at a timing when apredetermined advertizing moving image in the video content isreproduced (during or immediate after the reproduction), the CPU 10 mayreproduce the advertisement B related to the predetermined advertizingmoving image by the terminal device 7. Alternatively, at a timing when aproduct is displayed in the video content, the CPU 10 may reproduce theadvertisement B regarding the product by the terminal device 7. Asdescribed above, the CPU 10 may output advertisement information to theterminal device 7 such that advertisement information corresponding tothe specific content of the video content to be reproduced by the TV 2is reproduced by the terminal device 7. Owing to this, regarding theproduct or the like introduced in the video content reproduced by the TV2, detailed information can be provided to the user by the terminaldevice 7 immediately. Therefore, at a timing when the user wishes toview (obtain more detailed information of) the product, such detailedinformation can be provided to the user. This can improve theadvertizing effect of the advertisement B. According to the above, forexample, a continuation to the advertisement (commercial) included inthe video content can be presented by the advertisement informationdisplayed on the terminal device 7. Owing to this, the advertisement canbe made interesting to the user, which can improve the advertizingeffect of the advertisement information.

As the advertisement B, a web page may be displayed as well as themoving image, the still image or the like. For example, in the casewhere the video content to be displayed on the TV 2 includes a URL, theweb page of the URL may be displayed on the terminal device 7. Morespecifically, when the URL of a website of a sponsor is displayed in thecommercial included in the video content, a page of the website may bedisplayed on the terminal device 7. Owing to this, the work of the userof inputting the URL can be omitted, and also the advertizing effect ofthe web page can be improved. An operation of purchasing the productusing the website may be made possible. In this case, the specificcontent of the video content and the purchase screen can be associatedwith each other to provide a situation close to TV shopping, whichprovides a high advertizing effect. It may be made possible to inputinformation for purchasing a product by a touch operation. The terminaldevice 7 may have a built-in RFID reader or may be connected to an RFIDreader, and thus a card may be read for payment.

Timing information attached to the video content may be used todetermine whether the specific content of the video content reaches thepredetermined state or not. The timing information may be anyinformation which can make the determination. For example, the timinginformation is a flag indicating that the video content reaches thepredetermined state, or a time stamp attached to each of images includedin the video content. The CPU 10 refers to the timing information tomake the determination. This determination method allows the displaytiming of the advertisement information to be controlled and managedeasily on the side of the transmission server 91 by use of thereproduction control instruction. In other embodiments, thedetermination may be made by the game device 3 analyzing the images ofthe video content.

In the first embodiment, the transmission server 91 uses thereproduction control instruction so that the display timing of theadvertisement information is controlled in the game system 1 as shown inFIG. 15. In other embodiments, the transmission server 91 may controlthe display timing of the advertisement information as shown in FIG. 15without using the reproduction control instruction (e.g., by streamingtransmission or the like).

Returning to FIG. 14, in step S29 after step S28, the CPU 10 generatesan image including the advertisement information and an operation image.In the first embodiment, the image generated in step S29 is displayed onthe terminal device 7. The operation image is an image regarding theoperation performed on the operating unit of the terminal device 7(touch panel 52, buttons 54A through 54M, etc.). Specifically, theoperation image may be an image of a button displayed on the terminaldevice 7 in order to allow an operation to be performed on the touchpanel 52, or an image representing an operation performed by use of anyof the buttons of the terminal device 7 (e.g., image representing“determine by button A” or “cancel by button B”).

In the first embodiment, an image for performing an operation regardingthe reproduction of the video content by the TV 2 (reproductionoperation image) and/or an image for performing an operation on the TV 2(TV operation image) is displayed on the terminal device 7 as theoperation image. FIG. 16 shows an example of reproduction operationimage. As shown in FIG. 16, the terminal device 7 displays areproduction operation image 120 in addition to advertisementinformation 124. The reproduction operation image 120 includes a stopbutton 121, a reproduction button 122, and a pause button 123. The stopbutton 121 is an image representing an instruction to stop thereproduction of the video content. The reproduction button 122 is animage representing an instruction to reproduce the video content. Thepause button 123 is an image representing an instruction to pause thereproduction of the video content. In other embodiments, an imagerepresenting an instruction to fast-forward or reverse the reproductionof the video content, an image representing a bar usable to issue aninstruction to specify the reproduction position of the video content,or the like may be displayed as a reproduction operation image.

FIG. 17 shows an example of TV operation image. As shown in FIG. 17, aTV operation image 130 is displayed on the terminal device 7. Althoughnot shown in FIG. 17 for the purpose of making the figure easier to see,the advertisement information 124 is displayed below the TV operationimage 130 in the first embodiment. In other embodiments, theadvertisement information 124 may not be displayed, or the TV operationimage 130 may be displayed over the advertisement information 124 in asemitransparent state. As shown in FIG. 17, the TV operation image 130includes a power button 131, an input switch button 132, a volume upbutton 133, a volume down button 134, a channel increase button 135, achannel decrease button 136, and a return button 137. The buttons 131through 136 are each an image representing a control instruction on theTV 2. The power button 131 is an image representing an instruction toswitch ON/OFF the power of the TV 2. The input switch button 132 is animage representing an instruction to switch the input to the TV 2 (e.g.,switching the input between a mode of inputting and displaying a videoof a TV program and a mode of inputting and displaying an image from thegame device 3). The volume up button 133 is an image representing aninstruction to increase a sound volume of the TV 2. The volume downbutton 134 is an image representing an instruction to decrease the soundvolume of the TV 2. The channel increase button 135 is an imagerepresenting an instruction to switch the station of the TV 2 one by onein an ascending manner. The channel decrease button 136 is an imagerepresenting an instruction to switch the station of the TV 2 one by onein a descending manner. By use of each of the buttons 131 through 136,the user can perform an operation on the power, input switch, soundvolume and station selection of the TV 2. The return button 137 is animage representing an instruction to stop the display of the TVoperation image 130. In other embodiments, buttons provided in a remotecontrol of a general TV, for example, a button representing each ofchannels, a button representing an instruction to display a programguide, a button representing an instruction to record (in the case wherethe game device 3 or the TV 2 has a recording function) and the like maybe displayed on the terminal device 7, in addition to theabove-mentioned buttons.

In the first embodiment, the reproduction operation in step S23 is anoperation of touching any of the buttons 121 through 123 of thereproduction operation image 120. The TV operation in step S27 is anoperation of touching any of the buttons 131 through 136 of the TVoperation image 130. Namely, in the first embodiment, the user canperform a reproduction operation and a TV operation by use of the touchpanel 52 of the terminal device 7. An input for these operations may becontrolled at a timing during the reproduction of the video content.Specifically, the CPU 10 may partially or entirely control theseoperations at a predetermined timing while the video content is beingdisplayed. For example, at a timing when a predetermined video isdisplayed, the CPU 10 may control such that an operation of stopping thedisplay, fast-forwarding the display or the like is not accepted, orsuch that such a button is not displayed. Owing to such control, thevideo which the provider of the video content (the side of thetransmission server) wishes to show can be viewed by the user withoutfail. For example, in the case where the video content include acommercial, the CPU 10 can control such that the commercial can beviewed without being skipped. An instruction to restrict thereproduction operation or the TV operation may be performed by thetransmission server 91. Specifically, such an instruction may betransmitted from the transmission server 91 to the game system 1together with the video information, and the CPU 10 may restrict theoperation based on the received instruction.

In the first embodiment, in the case where a predetermined operation ofcausing the TV operation image 130 to be displayed is performed, the CPU10 generates an image including the advertisement information 124 andthe TV operation image 130 overlapped thereon (see FIG. 17). Bycontrast, in the case where the predetermined operation is notperformed, the CPU 10 generates an image including the advertisementinformation 124 and the reproduction operation image 120 (FIG. 16). Thepredetermined operation may be any operation. For example, thepredetermined operation may be an operation of pressing a predeterminedbutton provided in the terminal device 7. Owing to this, the user cancause the TV operation image 130 to be displayed easily by oneoperation, which is convenient.

In step S29, the process is executed as follows specifically. The CPU 10reads an image of the advertisement information to be displayed from theflash memory 17, and generates an image including the image and also thereproduction operation image 120 (and the TV operation image 130 whennecessary). Such an operation image may be generated in any manner. Inthe first embodiment, image data usable for generating the reproductionoperation image 120 and the TV operation image 130 is stored on the mainmemory or the like in advance. In other embodiments, the image data maybe acquired from the transmission server 91 together with the videoinformation as described later. The CPU 10 stores the generated image onthe storage unit of the game device 3 (e.g., flash memory 17, mainmemory, or VRAM 11 d). After step S29, the CPU 10 terminates thereproduction control process.

As described above regarding step S29, in the first embodiment, in thecase where the video content is displayed on the TV 2, the terminaldevice 7 displays the operation image together with the advertisementinformation (FIG. 16 and FIG. 17). Owing to this, the user views theoperation image when performing an operation on the video content.Therefore, it is possible to direct the user's attention also to theadvertisement information displayed on the terminal device 7 togetherwith the operation image. According to the process of step S29, theoperation image is displayed on the terminal device 7, and thus theadvertizing effect of the advertisement information can be improved.

In the first embodiment, the terminal device 7 includes the touch panel52, and the operation for performing control on the reproduction and/orthe display of the video content on the TV 2 is performed by use of thetouch panel 52. Therefore, the user can perform an operationperceptively and easily by use of the touch panel 52. When performing anoperation by use of the touch panel 52, the user views the operationimage. Therefore, it is possible to direct the user's attention to theadvertisement information displayed together with the operation image,with certainty. This can improve the advertizing effect of theadvertisement information.

In other embodiments, the reproduction operation may be partially orentirely set to be different in accordance with the transmission server91 as an addressee of the video content and/or in accordance with thevideo content transmitted. For example, the transmission server 91transmits information on the reproduction operation (informationrepresenting the correspondence between the content of the reproductionoperation and the operation to be performed by the game device 3 inaccordance with the reproduction operation) and data on the reproductionoperation image together with the video information, and the game device3 executes the process of step S25 using the information on thereproduction operation. The game device 3 also executes the process ofstep S29 using the received data on the reproduction operation image.Owing to this, the method of the reproduction operation can becontrolled on the side of the transmission server 91, and thus anoperation method suitable to the video content can be provided.

Returning to FIG. 12, in step S6 after step S5, the CPU 10 outputs thevideo content to the TV 2. First, the CPU 10 reads the content data 113stored on the flash memory 17, and stores an image included in themoving image of the video content on the VRAM 11 d. At this point, theCPU 10 and the GPU 11 b generate an image from the data stored on theflash memory 17 when necessary. For example, in the case where thecontent data 113 stored on the flash memory 17 is compressed by apredetermined scheme, the data on the moving image is decompressed togenerate an image. In the case where the data on the moving image isstored in units of packets, an image is generated from the data in unitsof packets. The image stored on the VRAM 11 d is transferred to theAV-IC 15, and the AV-IC 15 outputs the image to the TV 2 via the AVconnector 16. Owing to this, an image included in the moving image isdisplayed on the TV 2. A sound may be acquired from the transmissionserver 91 together with the moving image of the video content. In stepS5, a sound may be output to the TV 2 together with the moving image andmay be output from the speaker 2 a of the TV 2. After theabove-described process of step S6, the process of step S7 is executed.

In step S7, the CPU 10 outputs (transmits) the image of theadvertisement information to the terminal device 7. Specifically, thedata on the terminal image stored on the storage unit in step S29 istransferred to the codec LSI 27 by the CPU 10, and is compressed aspredetermined by the codec LSI 27. The compressed image data istransmitted to the terminal device 7 by the terminal communicationmodule 28 via the antenna 29. The terminal device 7 receives the imagedata, transmitted from the game device 3, by the wireless module 80, andthe received image data is decompressed as predetermined by the codecLSI 76. The decompressed image data is output to the LCD 51. Owing tothis, the terminal image is displayed on the LCD 51. After the processof step S7, the process of step S8 is executed.

The advertisement information including a sound together with the imagemay be acquired from the transmission server 91. In this case, in stepS7, the sound may be transmitted to the terminal device 7 together withthe image, and may be output from the speakers 77 of the terminal device7. In the case where information on the sound to be reproduced togetherwith the video content is included in the video information and isreproduced by the TV 2 together with the video content, the terminaldevice 7 may be controlled so as not to output the sound such that thereproduction by the TV 2 is not disturbed. Specifically, in this case,the advertisement information may be controlled so as not to include thesound information. In other embodiments, at the above-mentioned “timingwhen the video content to be reproduced by the TV 2 reaches apredetermined state (timing T1 shown in FIG. 15)”, the CPU 10 may notifythe user of the timing by use of the terminal device 7. The timing maybe notified to the user by, for example, outputting a sound from theterminal device 7 or by vibrating the vibrator 89 of the terminal device7.

In step S8, the CPU 10 determines whether or not to terminate thereproduction of the video content by the TV 2. The determination in stepS8 is made, for example, in accordance with whether the video contenthas been reproduced until the end or not, or in accordance with whetheran instruction to stop the reproduction of the video content has beenissued or not. When the determination result of step S8 is positive, theprocess of step S2 is executed again. Specifically, when thereproduction of the video content is terminated, the processing unit 102specifies the video content to be reproduced next in accordance with aninstruction by the user. By contrast, when the determination result ofstep S8 is negative, the process of step S9 is executed.

In step S9, the CPU 10 determines whether or not to terminate theviewing of the video content (and the advertisement information). Thedetermination in step S9 is made, for example, in accordance withwhether an instruction to terminate the viewing has been issued by theuser or not. When the determination result of step S9 is negative, theprocess of step S5 is executed again. After this, while the reproductionof the video content is continued (while the determination results ofsteps S8 and S9 are negative), the CPU 10 executes the processes ofsteps S5 through S9 in repetition. By contrast, when the determinationresult of step S9 is positive, the CPU 10 terminates the process shownin FIG. 12.

As described above, in the first embodiment, the video content includedin the video information transmitted from the transmission server 91 isdisplayed on the TV 2, and the advertisement information included in thevideo information is displayed on the terminal device 7. Owing to this,the advertisement information can be presented to the user effectively,and thus the advertizing effect can be improved.

In the case where a program which the user has missed is provided as thevideo content as in the first embodiment, it is possible to motivate theuser who has viewed such a program to view TV broadcasting related tothe program which he/she missed. In the case where a version for sampleviewing of the video content is provided as the advertisementinformation, it is possible to motivate the user to view the videocontent. In this manner, according to the first embodiment, regardlessof whether a TV program is provided as the video content (withoutcharge) or for a charge, a video distribution system advantageous to theprovider of the video content (broadcaster, etc.) can be constructed.

1-8. Modifications of the First Embodiment

(Modification in which Related Information is Transmitted)

In the above embodiment, the advertisement information is transmitted asinformation to be transmitted together with the video content. In otherembodiments, related information relating to the video content may betransmitted together with the video content, instead of theadvertisement information. The related information is, for example,information related to the information introduced in the video content,additional information related to the video content or the like.Specifically, in the case where the video content is a cooking program,information on a recipe of a dish may be displayed on the terminaldevice 7 as the related information. In the case where the video contentis a film or a TV drama, information on the cast may be displayed on theterminal device 7 as the related information. The related informationincludes advertisement information of a product introduced in the videocontent or advertisement information on the video content itself. In thecase where the video content includes an advertisement, advertisementinformation related to the advertisement can be the related information.Namely, such advertisement information may be displayed on the terminaldevice 7 as the related information. In the case where the relatedinformation is displayed on the terminal device 7 as described above,the related information can be displayed for the same time duration asthe reproduction time duration of the video content. Therefore, a largeamount of information regarding the video content can be presented tothe user. In the case where the related information is displayed on theterminal device 7, like in the case of the advertisement information,the related information displayed on the terminal device 7 is notdisplayed on the TV 2. Therefore, the related information does notdisturb the video content displayed on the TV 2, and thus the videocontent can be presented to the user in an easy-to-view manner.

(Modification in which Advertisement Information is Transmitted DuringTV Broadcasting)

In the above embodiment, in the case where the video content transmittedfrom the transmission server 91 is displayed on the TV 2, advertisementinformation is displayed on the terminal device 7. In other embodiments,in the case where a TV program received by TV broadcasting is displayedon the TV 2, advertisement information (or related information) may bedisplayed on the terminal device 7. In this case, it is conceivable thatthe video display system operates as follows.

The game device 3 first acquires program selection information usablefor selecting a TV program (e.g., information on the electronic programguide) from a predetermined server. In response to the acquisitionrequest from the game device 3, the server transmits the programselection information to the game device 3. Upon receipt of the programselection information, the game device 3 causes the terminal device 7 orthe TV 2 to display the program selection information. In accordancewith, for example, an operation performed by the user on the terminaldevice 7, the game device 3 specifies a TV program to be viewed among aplurality of TV programs included in an image of the program selectioninformation. The game device 3 also transmits information on thespecified TV program to the server as transmission information. Thetransmission information may include attribute information or currentinformation, like in the above embodiment. Upon receipt of thetransmission information, the server transmits advertisement informationto the game device 3. The advertisement information to be transmittedmay be selected in accordance with the attribute information and/or thecurrent information included in the transmission information like in theabove embodiment, or may be selected in accordance with the specified TVprogram. The server may transmit the advertisement information of acontent suitable to the TV program specified by the game device 3.Meanwhile, the game device 3, after specifying the TV program, controlsthe TV 2 such that the program is received and displayed. The gamedevice 3 also causes the advertisement information transmitted from theserver to be displayed on the terminal device 7. Owing to the above, thegame system 1 can display the TV program by the TV broadcasting on theTV 2 and also display the advertisement information on the terminaldevice 7.

(Modification in which a Plurality of Terminal Devices are Included)

In the above embodiment, the video display system 90 includes oneterminal device 7. In other embodiments, the video display system mayinclude a plurality of terminal devices 7. In the case where there are aplurality of terminal devices, the terminal devices may displaydifferent advertisement information and/or related information or thesame advertisement information and/or related information. For example,one terminal device may display advertisement information while anotherterminal device may display related information. In this manner, in thecase where there are a plurality of terminal devices, diversifiedinformation can be provided to the user.

In other embodiments, even in the case where the game system 1 is notused, a combination of the video content and the advertisement can bepresented in substantially the same manner as above. For example,information devices such as a set top box (receiving device) connectedto a TV and a tablet terminal (terminal device) may be used. In the caseof a structure using a set top box and a tablet terminal, the videocontent and the advertisement information included in the videoinformation may be respectively distributed to the set top box and thetablet terminal by a router or the like so that the set top box and thetablet terminal respectively display the video content and theadvertisement information. Alternatively, when synchronization isperformed, the video content and the advertisement information arereceived by the set top box and the tablet terminal respectively andreproduced in association with each other. Specifically, thetransmission server 91 may transmit the video information including thevideo content and the advertisement information to the set top box (TV2) and the tablet terminal via the network 92. The transmission server91 may transmit the video content and the advertisement information tothe respective display devices separately. Specifically, thetransmission server 91 may transmit the video content to the set top boxand transmit the advertisement information to the tablet terminal. TheTV 2 may display the received video content while the tablet terminalmay display the received advertisement information in association withthe display of the video content on the TV 2 (in the case where thevideo content is displayed on the TV 2).

2. Second Embodiment 2-1. Configuration of Information Processing System

Hereinafter, an example of information processing system according to asecond embodiment will be described. In the second embodiment, it isassumed that the video content to be reproduced by the TV 2 is, forexample, a video content provided by TV broadcasting, a video contentstored on the storage medium of the TV 2, or the like, which is notprovided from the transmission server 91. In the second embodiment, asound of the video content reproduced by the TV 2 is sensed by theterminal device 7, and sound information on the sensed sound istransmitted to the server. The server specifies the video contentreproduced by the TV 2 based on the received sound information, andtransmits presentation information corresponding to the video content tothe terminal side. A terminal device 7 displays the presentationinformation transmitted from the server. In this manner, in the secondembodiment, the video content reproduced by the TV 2 and alsoinformation suitable to the video content can be provided to the user.

In the second embodiment, the presentation information is theadvertisement information described in the first embodiment. Note thatthe presentation information is not limited to the advertisementinformation and may be any type of information. For example, thepresentation information may be the above-described related informationrelating to the video content. The presentation information can beconsidered to be additional information to the video content. Therefore,the presentation information may include information which is differentfrom the information included in the video content itself. Typically,the presentation information is information which is different fromidentification information of the video content (information on a titleor the like of the video content). The presentation information isdisplayable on a display device, and may be a moving image, a stillimage, or a web page. The presentation information may include soundinformation.

FIG. 18 is a block diagram showing an example of information processingsystem according to the second embodiment. As shown in FIG. 18, aninformation processing system 200 in the second embodiment includes aserver 201 and the game system 1. The server 201 and the game system 1are mutually communicable via the network 92. In the second embodiment,identical elements as those of the first embodiment bear identicalreference numerals thereto, and detailed descriptions thereof will beomitted.

In the second embodiment, the game system, which is an example ofterminal-side system (terminal system), has substantially the sameconfiguration as that in the first embodiment. However, processesexecuted by the game system 1 are different from those in the firstembodiment (described later in detail). In other embodiments, theterminal system may include one device. For example, the terminal systemmay be configured as a smart phone, a mobile phone, a tablet terminal, amobile game device or the like.

The server 201, which is an example of server-side system (serversystem), specifies the video content which is being reproduced on theside of the game system 1 based on information from the game system 1,and transmits the presentation information. FIG. 19 is a block diagramshowing an example of configuration of the server 201. As shown in FIG.19, the server 201 includes the communication unit 101, the processingunit 102, and the advertisement storage unit 104 which are substantiallythe same as those in the first embodiment. As described later in detail,processes executed by the processing section 102 in the secondembodiment are different from those in the first embodiment.

The server 201 also includes a database storage unit 202. The databasestorage unit 202 stores video database usable for specifying the videocontent to be reproduced on the terminal side. The video databaseincludes, for example, data associating the above-described soundinformation with the identification information of the video contentincluding a sound corresponding to the sound information. Theidentification information may be, for example, a title of the videocontent (title of a TV program, title of a film, etc.).

The database storage unit 202 stores a presentation information databaseusable for specifying the presentation information to be transmitted tothe terminal side. The presentation information database includes, forexample, data associating the identification information of the videocontent with one or more pieces of presentation information to bepresented together with the video content. In the second embodiment, thepresentation information database includes information which representsthe timing to present the presentation information, for each piece ofpresentation information. As described later in detail, such a databasemay be prepared in advance, or a specific content of the database may beadded or updated sequentially (in real time).

The server 201 may have any configuration which is communicable with anexternal device (game system 1, etc.) via the network 92 and has afunction of transmitting presentation information in response to arequest from the external device. The server 201 (server system) mayinclude one or more information processing devices. For example, theadvertisement storage unit 104 and the database storage unit 202 may beincluded in different servers. In this specification, in the case wherethe server system includes a plurality of server devices, the entiretyof the server devices will be referred to as a “server”.

2-2. Overview of Operation of Information Processing System

Now, with reference to FIG. 18, FIG. 20 and FIG. 21, an overview of anoperation of the information processing system 200 will be described.FIG. 20 shows an example of operation flow of the information processingsystem 200.

(Overall Process Flow of Information Processing System)

As shown in FIG. 20, in the second embodiment, the video content isreproduced by the TV 2. The video content may be any content, and maybe, for example, a program of TV broadcasting, or a video content storedon a storage medium (hard disc, DVD, etc.) accessible to the TV 2. Inthe second embodiment, an example in which a TV program which iscurrently being broadcast is reproduced by the TV 2 will be described.In the second embodiment, the video content does not need to be providedfrom the server, and therefore the terminal system may not include theTV 2 or may not be connected to (communicable with) the TV 2.

As shown in FIG. 18 and FIG. 20, in the case where the video content isreproduced by the TV 2, the terminal device 7 senses a reproductionsound of the video content by the microphone 79. The terminal device 7transmits the sound (microphone sound) sensed by the microphone 79 tothe game device 3. The game device 3 generates sound information basedon the microphone sound. The sound information is informationrepresenting the microphone sound or a feature of the microphone sound.The sound information is, for example, information on an acousticfingerprint. The sound information may be a sound signal itself of themicrophone sound. The terminal system (game system 1) transmits thegenerated sound information to the server 201.

As shown in FIG. 20, based on the received sound information, the server201 specifies the video content which is being reproduced in thevicinity of the terminal device 7. The specification of the videocontent is performed by use of the video database described above. Inthe second embodiment, the server 201 specifies a TV program which iscurrently being reproduced by the TV 2.

After specifying the video content which is being reproduced on theterminal side, the server 201 specifies the presentation information tobe presented together with the specified video content. Thespecification of the presentation information is performed by use of thevideo database described above. In the second embodiment, the server 201specifies the advertisement information to be presented together withthe video content. The server 201 transmits the specified presentationinformation (advertisement information) to the terminal system.

The terminal system displays the presentation information transmittedfrom the server 201 on the display unit (LCD 51). Specifically, as shownin FIG. 18, the game device 3 receives the presentation informationtransmitted from the server 201 and transmits the received presentationinformation to the terminal device 7. The terminal device 7 displays thereceived presentation information on the LCD 51. As described above, inthe second embodiment, the video content is reproduced by the TV 2, andalso the presentation information is displayed on the terminal device 7.Owing to this, information useful for the user can be presented, andalso other information can be presented to the user effectively togetherwith the video content.

(Determination on Presentation Timing)

In the second embodiment, the information processing system 200 maydetermine the timing to present the presentation information. Whenspecifying the video content based on the sound information, the server201 can specify (or estimate) the timing when a desired specific contentis to be reproduced if a specific content of the specified video content(TV program) is known to the server 201. In this case, the server 201can present the presentation information on the terminal side at thetiming when the desired content is reproduced. For example, in a timezone in which a product is introduced in a TV program, the informationprocessing system 200 can present the advertisement information of theproduct on the terminal device 7. For example, in a time zone in which acountry is introduced in a TV program, the information processing system200 can present the advertisement information on travelling in thecountry.

FIG. 21 shows an example of presentation timing of presentationinformation. In the second embodiment, the presentation timing isdetermined based on the video content specified by the server 201.Specifically, the server 201 first specifies the current position of thespecified video content on a time axis based on the sound information(see FIG. 21). The “current position” is the position of the time point(or time zone) when the sound sensed by the microphone 79 on theterminal side is reproduced. Next, the server 201 determines the timingto present the presentation information based on the current time and atime table of the specified video content. The time table is, forexample, information associating some scenes included in the videocontent with the time elapsed from the start of the video content. Forexample, a case where presentation information is displayed at thetiming when a scene A in the video content is reproduced will bediscussed. In this case, where the time elapsed from the start of thevideo content until the current time is time T2 and the time elapsedfrom the start of the video content until the start of the scene A isT3, the server 201 determines the timing after an elapse of time T4(=T3−T2) from the current time as the presentation timing. The time T2can be specified as the position of the current time based on the soundinformation. The time T3 can be specified based on the time table. Inthis manner, the server 201 can determine the presentation timing.

The terminal system displays the presentation information on the displayunit at the presentation timing determined as described above. Owing tothis, the presentation information can be presented at an appropriatetiming, and therefore can be presented to the user effectively. Thecontrol on reproducing the presentation information at the presentationtiming may be performed on the server side or on the terminal side.Specifically, in the case where the control is performed on the serverside, the server 201 may control a transmission timing of thepresentation information such that the presentation information isdisplayed on the display unit at the presentation timing. In this case,the server 201 may determine the transmission timing in consideration ofa delay caused by communication or the like. In the case where thecontrol is performed on the terminal side, the server 201 transmits thepresentation information and information representing the presentationtiming to the terminal system. This information may be, for example,information which instructs to control the reproduction described in thefirst embodiment. The terminal system controls the timing to present thepresentation information based on the information representing thepresentation timing.

(Control on TV)

As described above in the first embodiment, the terminal system cancontrol the operation of the TV 2. Specifically, the terminal system cancontrol a predetermined display device (TV 2) which can reproduce avideo content, regarding the reproduction and/or the display of thevideo content. Owing to this, the terminal system can change anoperation of the predetermined display device (e.g., change of thereceiving channel, stop, fast-forwarding or the like of thereproduction). Thus, the video content and the presentation informationcan be presented to the user effectively. For example, the terminalsystem can change the reproduction of the video content in accordancewith the reproduction of the presentation information. Therefore, it ispossible to attract attention of the user to the terminal device 7 onwhich the presentation information is displayed by, for example,decreasing the reproduction sound volume of the video content. In thesecond embodiment, the method for controlling the operation of the TV 2is substantially the same as that in the first embodiment.

As described above in the first embodiment, the terminal device 7 has afunction of controlling the operation of the TV 2. Therefore, the usercan change the operation of the TV 2 by use of the terminal device 7,and thus can improve the ease of operation on the video content. Theuser views the terminal device 7 when performing an operation on thevideo content. Therefore, at this point, it is possible to direct theuser's attention to also the presentation information displayed on theterminal device 7. Accordingly, in the case where the presentationinformation is advertisement information, the advertizing effect of theadvertisement information can be improved.

In the second embodiment, a control command for controlling the TV 2 maybe transmitted from the server 201. Specifically, the server 201specifies the control command on the reproduction and/or the display ofthe video content based on the specified video content, and transmitsthe control command to the terminal system. The terminal system controlsthe TV 2 based on the control command received from the server 201.Owing to this, the operation of the TV 2 can be controlled on the serverside. For example, the server 201 can control the operation of the TV 2such that the video content and the presentation information arepresented to the user effectively. For example, the following operationis made possible by transmission of the control command.

FIG. 22 shows an example of operation flow of the information processingsystem in the case where a control command is transmitted from theserver side. In the example shown in FIG. 22, it is assumed that thevideo content reproduced by the TV 2 is a TV program. It is also assumedthat after the broadcast of the TV program is finished at a certainbroadcast station (channel), a sequel video content to the TV program isbroadcast at another broadcast station in a continuous manner. In thiscase, when the program is finished at the certain broadcast station, theserver 201 generates a control command to change the channel (selectedstation) to the another broadcast station, and transmits the controlcommand to the terminal side. In the example of FIG. 22, this controlcommand is transmitted when the program is finished. Alternatively, thecontrol command may be transmitted before the program is finished. Uponreceiving the control command, the game device 3 transmits the controlcommand to the terminal device 7. The terminal device 7 outputs thecontrol command (e.g., an infrared or wireless signal corresponding tothe control command) to the TV 2. Owing to this, the channel of the TV 2is changed to the different broadcast station which broadcasts thesequel video content with no operation performed by the user. Thecontrol command is not limited to a command to change the channel whichis issued when the video content is finished, and may be any commandregarding the reproduction and/or the display of the video content.

2-3. Operation of Information Processing System

Now, an operation of the information processing system 200 will bedescribed in detail. First, various types of data usable for theoperation of the information processing system 200 will be described.FIG. 23 shows an example of data stored on a storage unit of the gamedevice 3 (including the external main memory 12, the internal mainmemory 11 e, and the flash memory 17). As shown in FIG. 23, a controlprogram 211, the operation data 111, microphone sound data 212, andprocessing data 213 are stored on the storage unit of the game device 3.In addition to the data shown in FIG. 23, various types of data usablefor a process executed by the game device 3 (image data, sound data andthe like) are stored on the storage unit. The data shown in FIG. 23 maybe stored on any other storage device accessible to the game device 3,as well as on the storage unit in the game device 3.

The control program 211 is a program for causing the CPU 10 to execute aprocess in the game device 3 (game system 1). In the second embodiment,the CPU 10 executes the control program 211, and thus processes of stepsshown in a flowchart shown in FIG. 24 (information presentation process)are executed. The control program 211 is partially or entirely read fromthe flash memory 17 at an appropriate timing after the game device 3 ispowered on, and thus is stored on the storage unit. The control program211 may be acquired from the optical disc 4 or another device externalto the game device 3 (e.g., via the Internet), instead of from the flashmemory 17.

The operation data 111 is substantially the same as that in the firstembodiment. The microphone data 212 is data representing a sound sensedby the microphone 79 (microphone sound). The microphone sound istransmitted from the terminal device 7, acquired by the game device 3,and stored on the storage unit as the microphone data 212.

The processing data 213 is data usable in the information presentationprocess (FIG. 24) described later. The processing data 213 includessound information data 214, presentation information data 215, and thecontrol command data 118 substantially the same as that in the firstembodiment.

The sound information data 214 is data representing the soundinformation described above. In the second embodiment, the soundinformation data 214 represents the acoustic fingerprint of themicrophone sound. The presentation information data 215 is datarepresenting the presentation information described above. The data onthe presentation information transmitted from the server 201 is storedon the storage unit as the presentation information data 215.

Now, with reference to FIG. 24 and FIG. 25, the information presentationprocess executed by the information processing system 200 in the secondembodiment will be described in detail. FIG. 24 is a flowchart showingan example of information presentation process flow executed by the gamedevice 3 in the second embodiment. In the second embodiment, the CPU 10executes the control program 211 and thus the information presentationprocess shown in FIG. 24 is executed.

The conditions under which the terminal system starts executing theinformation presentation process are arbitrary. The execution of theinformation presentation process may be started when, for example, theterminal system is powered on. Alternatively, the execution of theinformation presentation process may be started when the control program211 is instructed to be started by a selection operation performed on amenu screen displayed on the terminal device 7 and/or the TV 2. Stillalternatively, the execution of the information presentation process maybe started, for example, in the case where the user is estimated to beviewing the TV 2. Specifically, for example, the informationpresentation process may be executed in response to the terminal systemacquiring an electronic program guide from the server by the methoddescribed in the first embodiment. Alternatively, the informationpresentation process may be executed in response to an operation forcontrolling the TV 2 performed by use of the terminal device 7.

At the start of the information presentation process shown in FIG. 24,first, in step S31, the CPU 10 acquires the microphone sound.Specifically, the terminal device 7 senses a sound in the vicinitythereof by the microphone 79, and transmits data on the sensed sound(microphone sound) to the game device 3. The data on the microphonesound is transmitted repeatedly, for example, at an interval ofpredetermined time duration, and the game device 3 acquires the data onthe microphone sound repeatedly. The game device 3 stores the acquireddata on the storage unit as the microphone sound data 212. After stepS31, the process of step S32 is executed.

In step S32, the CPU 10 generates sound information based on themicrophone sound, and transmits the sound information to the server 201.The sound information is generated based on the microphone sound. In thesecond embodiment, the CPU 10 generates sound information (acousticfingerprint) from the microphone sound data 212 read from the storageunit. Data on the generated sound information is stored on the storageunit as the sound information data 214. The sound information data 214stored on the storage unit is transmitted to the server 201. A specificoperation of transmitting the data to the server 201 is substantiallythe same as that in the first embodiment. After step S32, the process ofstep S33 described later is executed.

When the sound information is transmitted to the server 201 in thismanner, the server 201 transmits the presentation information to theterminal system. Hereinafter, with reference to FIG. 25, the processexecuted by the server 201 will be described.

FIG. 25 is a flowchart showing an example of presentation informationtransmission process flow executed by the server 201 in the secondembodiment. The presentation information transmission process isexecuted by the processing unit 102 of the server 201 executing apredetermined control program. The process shown in FIG. 25 is started,for example, in response to an access made by the terminal system (gamesystem 1). When a plurality of terminal systems access the server 201,the processing unit 102 may execute the presentation informationtransmission process in parallel for the plurality of terminal systems.

At the start of the presentation information transmission process shownin FIG. 25, first, in step S41, the processing unit 102 receives thesound information. Next in step S42, the processing unit 102 specifiesthe video content based on the received sound information. Theprocessing unit 102 reads the video database described above from thedatabase storage unit 202, and refers to the video database to specifythe video content associated with the received sound information. Morespecifically, when the received sound information matches a part ofsound information included in the video database, the processing unit102 specifies the video content associated with this specific soundinformation in the video database. After step S42, the process of stepS43 is executed.

The video database may be generated or updated in real time. Forexample, a program of TV broadcasting is received by the server 201 andthe sound in the program is analyzed to generate sound information(acoustic fingerprint), so that the database can be dynamically updated.

In step S43, the processing unit 102 specifies the presentationinformation based on the specified video content. The processing unit102 reads the presentation information database described above from thedatabase storage unit 202, and specifies the presentation informationassociated with the specified video content in the presentationinformation database. In the case where there are a plurality of piecesof presentation information to be specified, the processing unit 102 mayset an order of presenting the presentation information. The order ofpresentation may be set randomly or set in advance in the presentationinformation database. After step S43, the process of step S44 isexecuted.

In step S44, the processing unit 102 determines the presentation timingbased on the specified presentation information. The determination ofthe presentation timing is performed by, for example, the methoddescribed above in the “(Determination on presentation timing)” section.The server 201 does not need to determine the presentation timing. Inthis case, the process of step S44 does not need to be executed. Afterstep S44, the process of step S45 is executed.

In step S45, the processing unit 102 generates a control command. Thecontrol command is generated based on the specified video content. Forexample, a database which associates video contents with controlcommands may be prepared, so that the control command is generated byreferring to the database. The server 201 does not need to transmit thecontrol command. In this case, the process of step S45 does not need tobe executed. After step S45, the process of step S46 is executed.

In step S46, the processing unit 102 transmits the presentationinformation and the like to the terminal system. In the secondembodiment, the presentation information and the control command aretransmitted. Specifically, the processing unit 102 reads thepresentation information (advertisement information) specified in stepS43 from the advertisement storage unit 104, and transmits thepresentation information to the terminal system. The reproductioncontrol instruction described in the first embodiment may be transmittedtogether with the presentation information. In the second embodiment,the processing unit 102 transmits the presentation information at thetransmission timing based on the presentation timing determined in stepS44. In the case where the presentation timing is not determined, thespecified presentation information may be transmitted sequentially. Thepresentation information may be transmitted by any method. In the secondembodiment, the server 201 performs streaming transmission like in thefirst embodiment.

The processing unit transmits the control command generated in step S45to the terminal system. The control command may be transmitted togetherwith the presentation information or may be transmitted at a differenttiming from the presentation timing.

The transmission of the presentation information and the like in stepS46 is continued until the transmission of the entirety of thepresentation information and the like is terminated. When thetransmission is completed, the processing unit 102 terminates thepresentation information transmission process shown in FIG. 25.

Returning to FIG. 24, in step S33, the game device 3 receives thepresentation information and the like. The received presentationinformation is stored on the storage unit as the presentationinformation data 215. In the case where the control command is received,the control command data 118 is stored on the storage unit. The processof receiving the presentation information (advertisement information)from the server 201 and storing the presentation information on thestorage unit is substantially the same as that in the first embodiment.After step S33, the process of step S34 is executed.

In step S34, the CPU 10 performs reproduction control on thepresentation information (advertisement information). The process ofstep S34 is performed in substantially the same manner as that in, forexample, the process of step S28 in the first embodiment (the processmay also include the process of step S29). As a result of the process ofstep S34, an image of the presentation information is displayed on thedisplay unit of the terminal device 7. After step S34, the process ofstep S35 is executed.

In step S35, the CPU 10 controls the TV 2 based on the control command.Specifically, the CPU 10 determines whether or not there is a controlcommand to be executed at the current time, in the control command data118 stored on the storage unit. For example, in the case where, at atiming corresponding to the timing when a control command is to beexecuted, the control command is transmitted from the server 201, theCPU 10 determines that the control command is to be executed at thetiming when the control command is received. In the case where theinformation representing the timing when a control command is to beexecuted is transmitted from the serve 201 together with the controlcommand, the CPU 10 makes a determination based on the information. Whenit is determined in step S35 that there is a control command to beexecuted at the current time, the CPU 10 controls the operation of theTV 2 based on the control command. A specific method of controlling theTV 2 is substantially the same as that in the first embodiment. Whenthere is no control command to be executed at the current time, the CPU10 terminates the process of step S35. After step S35, the process ofstep S36 is executed.

In step S36, the CPU 10 determines whether or not the reproduction ofthe video content by the TV 2 has been terminated or not. Thedetermination in step S36 may be performed in any manner. For example,the CPU 10 may make the determination by receiving the termination timeof the video content from the server 201 in advance. Alternatively, forexample, in the case where the server 201 performs streamingtransmission of the presentation information until the reproduction ofthe video content is terminated, the CPU 10 may determine that thereproduction of the video content has been terminated when thepresentation information is not received from the server 201 for apredetermined time duration. When the determination result of step S36is negative, the process of step S33 is executed again. After this,while the reproduction of the video content by the TV 2 is continued (aslong as the determination result of step S36 is negative), the CPU 10executes the series of processes of steps S33 through S36 in repetition.By contrast, when the determination result of step S36 is positive, theCPU 10 terminates the information presentation process shown in FIG. 24.

In other embodiments, the processes of steps S31 and S32 described abovemay be executed in repetition until the reproduction of the videocontent is terminated like the processes of steps S33 through S35.Namely, the terminal system may execute the acquisition of themicrophone sound and the transmission of the sound information inrepetition. When receiving the sound information from the terminalsystem again, the server 201 executes the process of specifying thevideo content again (step S42). When the specified video content ischanged, the server 201 specifies the presentation information again(step S43), and transmits the presentation information again (step S46).Owing to this, even in the case where the video content reproduced bythe TV 2 is changed in the middle (e.g., in the case where the channelis changed), presentation information suitable to the post-change videocontent can be provided to the user.

As described above, in the second embodiment, the presentationinformation (advertisement information) transmitted from the server 201is displayed on the terminal device 7 while the video content reproducedby the TV 2 is being viewed. Owing to this, information useful for theuser can be presented, and also the information can be presented to theuser effectively as being added to the video content.

2-4. Modifications of the Second Embodiment

(Modification in which a Video Content is Transmitted from the Server)

In the second embodiment described above, the video content reproducedby the TV 2 is not provided from the server 201. In a modification ofthe second embodiment, a video content provided from the server 201 maybe reproduced by the TV 2. Specifically, based on the video contentspecified as the video content which is being reproduced by the TV 2,the server 201 may specify a sequel video content and transmit thesequel video content to the terminal system. The “sequel video content(to the specified video content)” is, for example, as follows: in thecase where the specified video content is one episode of a drama series,the sequel video content (to the specified video content) is the nextepisode; or in the case where the specified video content is a film, thesequel video content (to the specified video content) is a sequel to thefilm.

FIG. 26 is a flowchart showing an example of presentation informationtransmission process flow executed by the server 201 in the modificationof the second embodiment. In FIG. 26, substantially the same steps asthose in FIG. 25 bear the same step numbers thereto, and detaileddescriptions thereof will be omitted.

As shown in FIG. 26, in this modification, after step S42, the processof step S51 is executed. In step S51, the processing unit 102 specifiesthe sequel video content based on the specified video content. Forexample, the server 201 prepares a video database which associates videocontents with sequel video contents to such video contents. Theprocessing unit 102 may specify the sequel video content by referring tothe video database. After the process of step S51 is terminated, theprocess of step S43 described above is executed.

In this modification, after step S46, the process of step S52 isexecuted. In step S52, the processing unit 102 transmits the sequelvideo content to the terminal system. A specific process of transmissionin step S52 may be substantially the same as that in step S15 or S43described above. After the process of step S52 is terminated, theprocessing unit 102 terminates the presentation information transmissionprocess.

FIG. 27 is a flowchart showing an example of information presentationprocess flow executed by the game device 3 in the modification of thesecond embodiment. In FIG. 27, substantially the same steps as those inFIG. 24 bear the same step numbers thereto, and detailed descriptionsthereof will be omitted.

In this modification, when the determination result of step S36described above is positive, the process of step S55 is executed. Instep S55, the CPU 10 receives the sequel video content from the server201. A specific process of receiving in step S55 may be substantiallythe same as that in step S22 or S33 described above. In FIG. 27, theprocess of step S55 is executed after the reproduction of the firstvideo content is terminated (after the process of step S36).Alternatively, the process of step S55 may be executed during thereproduction of the first video content (during the execution of theprocessing loop of steps S33 through S35). After step S55, the processof step S56 is executed.

In step S56, the CPU 10 reproduces the sequel video content by, forexample, substantially the same method as that in step S6 and/or S7. Thesequel video content may be reproduced by the TV 2, by the terminaldevice 7, or both thereof. In the case where the sequel video content isreproduced by the TV 2, the presentation information may be reproducedby the terminal device 7.

According to this modification, after the reproduction of the videocontent by the TV 2 is terminated, the information processing system canprovide the sequel video content to such a video content to the user.Owing to this, information useful for the user can be presented.

(Modification in which Statistics Information of the Video Content isGenerate)

In another modification of the second embodiment, the server 201 maygenerated statistics information on the specified video content. Thestatistics information is obtained from a history of video contentsspecified in a predetermined time duration in the past. A specificexample of the statistics information may be information representingthe viewing frequency classified by genre of the video contents,information representing the viewing frequency classified by broadcaststation, or the like.

FIG. 28 is a flowchart showing an example of presentation informationtransmission process flow executed by the server 201 in the anothermodification of the second embodiment. In FIG. 28, substantially thesame steps as those in FIG. 25 bear the same step numbers thereto, anddetailed descriptions thereof will be omitted.

As shown in FIG. 28, in this modification, after step S42, the processof step S61 is executed. In step S61, the processing unit 102 generatesstatistics information based on the history of the specified videocontent. In this modification, in step S42, the processing unit 102stores the information on the specified video content as a history.Based on the history, the statistics information is generated. Afterstep S61, the process of step S43 is executed.

In this modification, in step S46, the processing unit 102 may transmitthe statistics information as being included in the presentationinformation to the terminal system. In this case, the statisticsinformation can be presented to the user. In step S43, the processingunit 102 may specify the presentation information based on thestatistics information. In this case, the presentation information canbe specified with the viewing history of the video content in the pastbeing reflected, and presentation information suitable to the user canbe specified.

(Modification in which a Sound Other than the Sound in the Video Contentis Sensed)

In the second embodiment described above, the server 201 executes theprocess of specifying the video content for the purpose of sensing thesound of the video content reproduced in the vicinity of the terminalsystem (terminal device 7). In other embodiments, the terminal systemmay execute the process for the purpose of sensing a sound other thanthe sound in the video content. For example, the terminal system maysense a voice of conversation of the user and transmit the soundinformation to the server 201. In this case, the server 201 may specifya specific content of the conversation from the sound information. Forexample, the server 201 may recognize some words during theconversation. Then, the server 201 may specify the presentationinformation corresponding to the specified content and transmit suchpresentation information to the terminal system. Owing to this, usefulinformation corresponding to the conversion of the user can be presentedto the user as well as the information added to the video content.

(Modification in which a Video Content is Specified)

In the second embodiment described above, the server 201 specifies thevideo content which is being reproduced by a predetermined displaydevice (TV 2) on the terminal side, by use of the sound information ofthe microphone sound. In other embodiments, the video content may bespecified by another method. For example, the terminal system maytransmit operation information representing an operation performed on apredetermined display device (TV 2) by use of the terminal device 7 tothe server 201. In this case, the server 201 may specify the videocontent based on the operation information. Specifically, in the casewhere an operation of changing the channel of the TV 2 is performed byuse of the terminal device 7, the terminal system transmits theoperation information representing the channel change operation to theserver 201. The server 201 specifies the TV program of the post-changechannel from the channel change operation, and transmits thepresentation information corresponding to the specified TV program tothe terminal system. Owing to this also, like in the second embodiment,presentation information suitable to the video content can be providedto the user.

(Modification Regarding the Configuration of the Terminal System)

In the second information, the server performs the process of specifyingthe video content from the sound information. In other embodiments, theabove-described process may be performed on the terminal side. Forexample, the terminal system may acquire (a part of) the video databasefrom the server and specify the video content by use of the acquiredvideo database. In this case, the process of specifying the presentationinformation may be performed on the server side or the terminal side. Inthe case where the process of specifying the presentation information isperformed on the server side, the terminal system may transmitidentification information of the video content to the server, and theserver may specify the presentation information based on theidentification information. In the case where the process of specifyingthe presentation information is performed on the terminal side, theterminal system may acquire (a part of) the presentation informationdatabase from the server and specify the presentation information by useof the acquired presentation information database.

In the first and second embodiments described above, a series ofinformation processes (FIG. 12 or FIG. 24) to be executed by theterminal system (game system 1) is executed by the game device 3. A partof such information processes may be executed by another device. Forexample, in other embodiments, a part of the information processes(e.g., generation of the terminal image) or the entirety thereof may beexecuted by the terminal device 7. In the case where the entirety of theinformation processes is executed by the terminal device 7, the terminaldevice 7 may be communicable with the transmission server 91 while thegame system 1 may not include the game device 3. In still otherembodiments, in a system including a plurality of information processingdevices communicable with each other via a network (wide area networkand/or local network), the plurality of information processing devicesmay execute the information processes in a shared manner. For example,the information processes maybe executed by a cloud-type ordistributed-type information processing system.

In the case where the plurality of information processing devicesexecute the information processes, the processes to be executed by theinformation processing devices are synchronized. Therefore, theinformation processes become complicated. By contrast, in the case wherethe information processes are executed by one game device 3 and theterminal device 7 executes the process of receiving and displaying theimage (namely, in the case where the terminal device 7 acts as a thinclient terminal) as in the first and second embodiments, there is noneed to synchronize the information processes among the plurality ofinformation processing devices and thus the information processes can besimplified. In the case where there is the game system 1 using the TV 2and the terminal device 7, the game system 1 can be applied to view thevideo content and the advertisement by the first embodiment. Thus, thesystem can be provided easily.

In the second embodiment, like in the first embodiment, the terminalsystem may include a plurality of terminal devices 7. In the case wherethere are a plurality of terminal devices, the terminal devices maydisplay different presentation information or the same presentationinformation. In the case where there are a plurality of terminaldevices, diversified information can be provided to the user.

The systems, devices and apparatuses described herein may include one ormore processors, which may be located in one place or distributed in avariety of places communicating via one or more networks. Suchprocessor(s) can, for example, use conventional 3D graphicstransformations, virtual camera and other techniques to provideappropriate images for display. By way of example and withoutlimitation, the processors can be any of: a processor that is part of oris a separate component co-located with the stationary display and whichcommunicates remotely (e.g., wirelessly) with the movable display; or aprocessor that is part of or is a separate component co-located with themovable display and communicates remotely (e.g., wirelessly) with thestationary display or associated equipment; or a distributed processingarrangement some of which is contained within the movable displayhousing and some of which is co-located with the stationary display, thedistributed portions communicating together via a connection such as awireless or wired network; or a processor(s) located remotely (e.g., inthe cloud) from both the stationary and movable displays andcommunicating with each of them via one or more network connections; orany combination or variation of the above.

The processors can be implemented using one or more general-purposeprocessors, one or more specialized graphics processors, or combinationsof these. These may be supplemented by specifically-designed ASICs(application specific integrated circuits) and/or logic circuitry. Inthe case of a distributed processor architecture or arrangement,appropriate data exchange and transmission protocols are used to providelow latency and maintain interactivity, as will be understood by thoseskilled in the art.

Similarly, program instructions, data and other information forimplementing the systems and methods described herein may be stored onone or more on-board and/or removable memory devices. Multiple memorydevices may be part of the same device or different devices, which areco-located or remotely located with respect to each other.

As described above, the above-described embodiments are usable for thepurpose of, for example, presenting information useful for the user, andare usable as, for example, an information processing system or the likefor providing information to be presented together with a video content,from the side of the server to the side of the terminal.

While certain example systems, methods, devices and apparatuses havebeen described herein, it is to be understood that the appended claimsare not to be limited to the systems, methods, devices and apparatusesdisclosed, but on the contrary, are intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

What is claimed is:
 1. An information processing system, comprising aterminal system which includes a terminal device and also comprising aserver system; wherein the terminal system includes: a microphoneprovided in the terminal device for sensing a sound in the vicinitythereof; a transmission unit for transmitting sound informationrepresenting the sound sensed by the microphone or a feature of thesound to the server system; and a display unit provided in the terminaldevice; the server system includes: a content specification unit forspecifying a video content which is being reproduced in the vicinity ofthe terminal device, based on the sound information; and a presentationinformation specification unit for specifying presentation informationto be presented together with the specified video content, in accordancewith the video content, and transmitting the presentation information tothe terminal system; and the terminal system displays the presentationinformation transmitted from the server system on the display unit. 2.The information processing system according to claim 1, wherein thepresentation information specification unit specifies, as thepresentation information, advertisement information on a product or aservice which is regarding the video content and/or which appears in thevideo content.
 3. The information processing system according to claim1, wherein: the server system further includes a timing determinationunit for determining a timing to present the presentation informationbased on the specified video content; the terminal system displays thepresentation information on the display unit at the determined timing.4. The information processing system according to claim 3, wherein thetiming determination unit specifies, based on the sound information, atime point of the sound sensed by the microphone on a time axis of thespecified video content, and determines the timing to present thepresentation information based on the time point and a time table of thespecified video content.
 5. The information processing system accordingto claim 1, wherein the terminal system further includes a devicecontrol unit for controlling a predetermined display device capable ofreproducing the video content, regarding reproduction and/or display ofthe video content.
 6. The information processing system according toclaim 5, wherein the device control unit is provided in the terminaldevice.
 7. The information processing system according to claim 5,wherein: the server system further includes an instruction transmissionunit for specifying, based on the specified video content, a controlcommand regarding the reproduction and/or the display of the videocontent, and transmitting the control command; and the device controlunit controls the predetermined display device based on the controlcommand received from the server system.
 8. The information processingsystem according to claim 1, wherein the server system further includesa statistics generation unit for generating statistics information onthe specified video content regarding the terminal system, andtransmitting the statistics information to the terminal system.
 9. Theinformation processing system according to claim 1, wherein: the serversystem further includes a content transmission unit for specifying,based on the specified video content, a sequel video content to thevideo content, and transmitting the sequel video content to the terminalsystem; and the terminal system causes a display device of the terminaldevice and/or a predetermined display device separate from the terminaldevice to display the sequel video content.
 10. An informationprocessing system, comprising a terminal system which includes aterminal device and an information processing device, and alsocomprising a server system; wherein the terminal device includes: amicrophone for sensing a sound in the vicinity thereof; and a displayunit; the information processing device include a transmission unit fortransmitting sound information representing the sound sensed by themicrophone or a feature of the sound to the server system; the serversystem includes: a sound specification unit for specifying the sound inthe vicinity of the terminal device based on the sound information; anda presentation information specification unit for specifyingpresentation information to be presented by the terminal device inaccordance with the specified sound, and transmitting the presentationinformation to the information processing device; and informationprocessing device receives the presentation information transmitted fromthe server system and causes the display unit of the terminal device todisplay the presentation information.
 11. The information processingsystem according to claim 10, wherein the sound specification unitspecifies a video content which is being reproduced in the vicinity ofthe terminal device.
 12. The information processing system according toclaim 11, wherein the presentation information specification unitspecifies, as the presentation information, advertisement information ona product or a service which is regarding the video content and/or whichappears in the video content.
 13. The information processing systemaccording to claim 11, wherein: the server system further includes atiming determination unit for determining a timing to present thepresentation information based on the specified video content; theterminal system displays the presentation information on the displayunit at the determined timing.
 14. The information processing systemaccording to claim 13, wherein the timing determination unit specifies,based on the sound information, a time point of the sound sensed by themicrophone on a time axis of the specified video content, and determinesthe timing to present the presentation information based on the timepoint and a time table of the specified video content.
 15. Theinformation processing system according to claim 11, wherein theterminal system further includes a device control unit for controlling apredetermined display device capable of reproducing the video contentregarding reproduction and/or display of the video content.
 16. Theinformation processing system according to claim 15, wherein the devicecontrol unit is provided in the terminal device.
 17. The informationprocessing system according to claim 15, wherein: the server systemfurther includes an instruction transmission unit for specifying, basedon the specified video content, a control command regarding thereproduction and/or the display of the video content, and transmittingthe control command; and the device control unit controls thepredetermined display device based on the control command received fromthe server system.
 18. The information processing system according toclaim 11, wherein the server system further includes a statisticsgeneration unit for generating statistics information on the specifiedvideo content regarding the terminal system, and transmitting thestatistics information to the terminal system.
 19. The informationprocessing system according to claim 11, wherein: the server systemfurther includes a content transmission unit for specifying, based onthe specified video content, a sequel video content to the videocontent, and transmitting the sequel video content to the terminalsystem; and the terminal system causes a display device of the terminaldevice and/or a predetermined display device separate from the terminaldevice to display the sequel video content.
 20. An informationprocessing system, comprising a terminal system which includes aterminal device and also comprising a server system; wherein: theterminal system includes: a microphone provided in the terminal devicefor sensing a sound in the vicinity thereof; a transmission unit fortransmitting sound information representing the sound sensed by themicrophone or a feature of the sound to the server system; and a displayunit provided in the terminal device; the server system includes: asound specification unit for specifying the sound in the vicinity of theterminal device based on the sound information; and a presentationinformation specification unit for specifying presentation informationto be presented by the terminal device based on the specified sound, andtransmitting the presentation information to the information processingdevice; and the terminal system receives the presentation informationtransmitted from the server system and causes the display unit of theterminal device to display the presentation information.
 21. A terminalsystem, including a terminal device, the terminal system comprising: amicrophone provided in the terminal device for sensing a sound in thevicinity thereof; a transmission unit for transmitting sound informationrepresenting the sound sensed by the microphone or a feature of thesound to a server system; a receiving unit for receiving, from theserver system, presentation information specified in accordance with thevideo content which is specified based on the sound information by theserver system; and a display unit provided in the terminal device fordisplaying the presentation information transmitted from the serversystem.
 22. A server system communicable with a terminal systemincluding a terminal device, the server system comprising: a receivingunit for receiving, from the terminal system, sound informationrepresenting a sound in the vicinity of the terminal device which issensed by a microphone or a feature of the sound; a contentspecification unit for specifying a video content which is beingreproduced in the vicinity of the terminal device based on the soundinformation; and a presentation information specification unit forspecifying presentation information to be presented together with thespecified video content, in accordance with the video content, andtransmitting the presentation information to the terminal system.
 23. Anon-transitory computer-readable storage medium having stored thereon aninformation processing program executable by a computer of a terminalsystem including a terminal device, the information processing programcausing the computer to execute: acquiring a sound in the vicinity ofthe terminal device which is sensed by a microphone provided in theterminal device; transmitting sound information representing the soundsensed by the microphone or a feature of the sound to a server system;receiving, from the server system, presentation information specified inaccordance with a video content which is specified based on the soundinformation by the server system; and causing a display unit provided inthe terminal device to display the presentation information transmittedfrom the server system.
 24. An information presentation methodexecutable by an information processing system comprising a terminalsystem which includes a terminal device and also comprising a serversystem, wherein: the terminal system: senses a sound in the vicinity ofthe terminal device by a microphone provided in the terminal device; andtransmits sound information representing the sound sensed by themicrophone or a feature of the sound to the server system; the serversystem: specifies a video content which is being reproduced in thevicinity of the terminal device, based on the sound information; andspecifies presentation information to be presented together with thespecified video content, in accordance with the video content, andtransmits the presentation information to the terminal system; and theterminal system displays the presentation information transmitted fromthe server system on a display unit.